{"id":34213,"date":"2017-12-14T12:39:32","date_gmt":"2017-12-14T04:39:32","guid":{"rendered":"http:\/\/localhost\/help\/?page_id=34213"},"modified":"2018-10-12T16:14:38","modified_gmt":"2018-10-12T08:14:38","slug":"data-transformation-and-sample-methods-4-3-0","status":"publish","type":"page","link":"https:\/\/oihelp.corporate.ifs.com\/help\/p2-server\/primers\/transforms\/data-transformation-and-sample-methods-4-3-0\/","title":{"rendered":"Data Transformation and Sample Methods (4.3.0)"},"content":{"rendered":"\n

This article applies to versions 4.3.0-4.5.3 of P2 Server. For the latest, see\u00a0Data Transformation and Sample Methods<\/a>.<\/p>\n

In P2 Server<\/a>, data may be requested raw or processed in some way. Currently the three kinds of data processing available are average<\/em>, last known value<\/em>, and linear interpolate<\/em>. In P2 Server these different aggregations are known as sample methods<\/em>.<\/p>\n

How each of these sample methods is determined can vary from one source of data to another.\u00a0 Most Historians already have these functions built in, so P2 Server delegates these aggregations to the Historian to perform, so that we get the fastest and most accurate aggregations possible.<\/p>\n

For any systems that do not have such features, P2 Server has a built-in set that will be used. \u00a0This article describes how these functions process the data.<\/p>\n

Background<\/h2>\n

Before we get into how these methods work, let\u2019s define some of the other inputs to these functions.\u00a0When we ask P2 Server for data we need to specify a few parameters so that P2 Server knows what to do with the data.<\/p>\n\n\n\n\n\n
\n

Start and End time (Request Interval)<\/strong><\/p>\n<\/td>\n

\n

The start and end time for the request is sometimes called the request interval<\/em>. These two times specify the whole range over which we want data. Note that start and end times are inclusive. For example, if we want data for the whole day, we may use 1\/1\/2015 00:00 and the 1\/1\/2015 23:59 to specify that we want data for that day only. Note that the calling application (e.g. P2 Explorer<\/a>) must send the request interval to P2 Server in UTC.<\/p>\n

When getting data from P2 Server, depending on how you set the request interval, you can get P2 Server to return different numbers of results. These two different methods are known as Range Fetches and Single Point (or Spot) Fetches.<\/p>\n<\/td>\n<\/tr>\n

\n

Sample Interval (Density)<\/strong><\/p>\n<\/td>\n

\n

The sample interval<\/em> is the amount of time between each data point that will be returned. If we set the sample interval to 3600 seconds, we will get a data point returned every hour between the start and end time (request interval). If that start and end time are 1 day apart, we will get 24 values returned, one per hour.<\/p>\n<\/td>\n<\/tr>\n

\n

Sample Methods<\/strong><\/p>\n<\/td>\n

\n

The rest\u00a0of this article details how the built-in aggregations work inside P2 Server.\u00a0Note that if the source system natively supports any of the following sample methods, adaptors will often be written to use the source system\u2019s functions, rather than P2 Server\u2019s calculations as described below. However this is dependent on the adaptor itself.<\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Related: Single Point Fetch vs Range Fetch<\/a>, Datasources and Adaptors<\/a><\/p>\n

Formulas<\/h2>\n

Average<\/h3>\n

The Average sample method returns a time-weighted average of the raw data for each sample interval over the specified request interval. The set of time-value pairs returned are generated using the following formula:<\/p>\n

\"image001\"<\/a><\/p>\n

The request interval [ts, tf]<\/strong> is divided into intervals using the configured sample interval SI<\/strong><\/em>. If the specified request interval is not an integral multiple of SI<\/strong><\/em>, then the returned results will cover slightly less than the desired interval (this gap is the interval [S2, tf]<\/strong>.<\/p>\n

As a result, the request interval [ts, tf]<\/strong> will be broken into N intervals, where:<\/p>\n

\"image003\"<\/a><\/p>\n

There will be N<\/strong><\/em> aggregate values returned at times (s0, s1, \u2026 , sN), where:<\/p>\n

\"image004\"<\/a><\/p>\n

A time-weighted average is calculated for each interval, which factors into the result the amount of time each raw value was held during the sample interval.<\/p>\n

In the above figure, the result for the second interval (s1,s2)<\/strong> will be v4<\/strong>, since the value is constant over the entire interval. As the raw value changes during (s0,s1)<\/strong>, the reported value will be an average of the four raw values over that interval, weighted by the duration that each raw value is held:<\/p>\n

\"image005\"<\/a><\/p>\n

The data used to calculate an average value for s0<\/strong> is based on an interval of SI <\/em><\/strong>that ends at s0<\/strong>. To fulfil a request for average-sampled data at a particular instant in time (ti<\/em>)<\/strong>, the adaptor <\/a>examines the raw data over the interval (ti-SI, ti)<\/strong> and calculates the time-weighted average value over that interval. The value is reported for time ti<\/strong><\/em>.<\/p>\n

 <\/p>\n

Last Known Value<\/h3>\n

The calculations of last known value<\/em> data returns the last known raw value at the end of each sampling interval. As with time-weighted average data, the request interval [ts,tf]<\/strong> is first divided into N<\/strong> sample intervals, and values are reported for the end of each interval. The set of reported values are simply the raw value of the input at each of the times (s0, s1, \u2026 , sN)<\/strong>.<\/p>\n

\"image006\"<\/a><\/p>\n

In the above figure, the returned data would be three points: (s0,v1)<\/strong>, (s1,v4)<\/strong> and (s2,v4)<\/strong>. For last-known-value at a single-point instant, the algorithm looks for the most recent raw value from the source, but returns it with a timestamp of the requested sample interval time, which will be the same or after the raw value's actual time (the time of s0, S1 and S2).<\/p>\n

 <\/p>\n

Linear Interpolate<\/h3>\n

For linear interpolate aggregate data, the sample period (ts, tf)<\/strong> is divided into intervals using the same process as for time-weighted average and last known value.<\/p>\n

\"image007\"<\/a><\/p>\n

Sample values are reported at the start of the request period, and all sample interval boundaries: (s0, s1, \u2026, sN). For each sample point, the formula looks for the most recent raw value and the closest subsequent raw value, and uses those two values as end points of a line. The value of that line at the sample point is the returned aggregate value.<\/p>\n

In the above figure, the value at s0<\/strong> is based on the nearest previous value at t1<\/strong> and the nearest subsequent value at t2<\/strong>. The value reported at s0<\/strong> would be:<\/p>\n

\"image009\"<\/a><\/p>\n

In some situations, such as the values for s2<\/strong> and s3<\/strong>, the same two raw value data points may be used to calculate linear interpolated values for two different sample points. For the last sample point, the formula will need a raw value potentially beyond the end of the requested sample period (after tf<\/strong>). The adaptor looks forward in time using configuration values to find the next raw value.<\/p>\n

Cheat Sheet<\/h2>\n

The following images provide a simplified overview of how the sample methods are calculated.<\/p>\n

\n

<\/i> \u00a0\u00a0Download the cheat sheet!<\/a><\/p>\n<\/div>\n

Average<\/h3>\n

\"\"<\/a><\/p>\n

Last Known Value<\/h3>\n

\"\"<\/a><\/p>\n

Linear Interpolate<\/h3>\n

\"\"<\/a><\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

In P2 Server, data may be requested raw or processed in some way. Currently the three kinds of data processing available are average, last known value, and linear interpolate. In P2 Server these different aggregations are known as sample methods. This article looks at the different sample methods available in P2 Server, and how they are calculated.<\/p>\n

Read more<\/i><\/a><\/p>\n","protected":false},"author":1,"featured_media":33928,"parent":3672,"menu_order":1,"comment_status":"open","ping_status":"closed","template":"","meta":{"footnotes":"","_members_access_role":[],"_members_access_error":""},"categories":[3],"tags":[276,285,100,291,292,296,289,290,212,288,287,154,422,277],"class_list":["post-34213","page","type-page","status-publish","has-post-thumbnail","hentry","category-primer","tag-aggregation","tag-average","tag-calculations","tag-data-fetch","tag-density","tag-formulas","tag-last-known-value","tag-linear-interpolate","tag-raw","tag-request-interval","tag-sample-methods","tag-time","tag-time-weighted","tag-transformation","Version-4-5-0","Version-4-5-1","Version-4-5-2","Version-4-5-3","Product-srv"],"_links":{"self":[{"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/pages\/34213","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/comments?post=34213"}],"version-history":[{"count":0,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/pages\/34213\/revisions"}],"up":[{"embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/pages\/3672"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/media\/33928"}],"wp:attachment":[{"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/media?parent=34213"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/categories?post=34213"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/oihelp.corporate.ifs.com\/help\/wp-json\/wp\/v2\/tags?post=34213"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}