Package org.rrd4j.data

Class DataProcessor

  • All Implemented Interfaces:
    DataHolder
    public class DataProcessor
extends Object
implements DataHolder

    Class which should be used for all calculations based on the data fetched from RRD files. This class supports ordinary DEF datasources (defined in RRD files), CDEF datasources (RPN expressions evaluation), SDEF (static datasources - extension of Rrd4j) and PDEF (plottables, see Plottable for more information.

    Typical class usage:

     final long t1 = ...
 final long t2 = ...
 DataProcessor dp = new DataProcessor(t1, t2);
 // DEF datasource
 dp.addDatasource("x", "demo.rrd", "some_source", "AVERAGE");
 // DEF datasource
 dp.addDatasource("y", "demo.rrd", "some_other_source", "AVERAGE");
 // CDEF datasource, z = (x + y) / 2
 dp.addDatasource("z", "x,y,+,2,/");
 // ACTION!
 dp.processData();
 // Dump calculated values
 System.out.println(dp.dump());

    • Field Detail

      • DEFAULT_PERCENTILE

        public static final double DEFAULT_PERCENTILE
        Constant DEFAULT_PERCENTILE=95.0
        See Also:
        Constant Field Values

    • Constructor Detail

      • DataProcessor

        public DataProcessor​(long t1,
                     long t2)
        Creates new DataProcessor object for the given time span. Ending timestamp may be set to zero. In that case, the class will try to find the optimal ending timestamp based on the last update time of RRD files processed with the processData() method.
        Parameters:
        t1 - Starting timestamp in seconds without milliseconds
        t2 - Ending timestamp in seconds without milliseconds

      • DataProcessor

        public DataProcessor​(Date d1,
                     Date d2)
        Creates new DataProcessor object for the given time span. Ending date may be set to null. In that case, the class will try to find optimal ending date based on the last update time of RRD files processed with the processData() method.
        Parameters:
        d1 - Starting date
        d2 - Ending date

      • DataProcessor

        public DataProcessor​(Calendar gc1,
                     Calendar gc2)
        Creates new DataProcessor object for the given time span. Ending date may be set to null. In that case, the class will try to find optimal ending date based on the last update time of RRD files processed with the processData() method. It use the time zone for starting calendar date.
        Parameters:
        gc1 - Starting Calendar date
        gc2 - Ending Calendar date

      • DataProcessor

        public DataProcessor​(java.time.temporal.TemporalAmount d)
        Creates new DataProcessor object for the given time duration. The given duration will be substracted from current time.
        Parameters:
        d - duration to substract.

    • Method Detail

      • isPoolUsed

        public boolean isPoolUsed()
        Returns boolean value representing RrdDbPool usage policy.
        Specified by:
        isPoolUsed in interface DataHolder
        Returns:
        true, if the pool will be used internally to fetch data from RRD files, false otherwise.

      • setPoolUsed

        public void setPoolUsed​(boolean poolUsed)
        Sets the RrdDbPool usage policy.
        Specified by:
        setPoolUsed in interface DataHolder
        Parameters:
        poolUsed - true, if the pool should be used to fetch data from RRD files, false otherwise.

      • setPixelCount

        public void setPixelCount​(int pixelCount)

        Sets the number of pixels (target graph width). This number is used only to calculate pixel coordinates for Rrd4j graphs (methods getValuesPerPixel(String) and getTimestampsPerPixel()), but has influence neither on datasource values calculated with the processData() method nor on aggregated values returned from getAggregates(String) and similar methods. In other words, aggregated values will not change once you decide to change the dimension of your graph.

        Parameters:
        pixelCount - The number of pixels. If you process RRD data in order to display it on the graph, this should be the width of your graph.

      • getPixelCount

        public int getPixelCount()
        Returns the number of pixels (target graph width). See setPixelCount(int) for more information.
        Returns:
        Target graph width

      • setStep

        public void setStep​(long step)
        Once data are fetched, the step value will be used to generate values. If not defined, or set to 0, a optimal step will be calculated.
        Specified by:
        setStep in interface DataHolder
        Parameters:
        step - Default to 0.

      • getStep

        public long getStep()
        Returns the time step used for data processing. Initially, this method returns zero. Once processData() is finished, the method will return the time stamp interval.
        Specified by:
        getStep in interface DataHolder
        Returns:
        Step used for data processing.

      • getFetchRequestResolution

        public long getFetchRequestResolution()
        Returns desired RRD archive step (resolution) in seconds to be used while fetching data from RRD files. In other words, this value will used as the last parameter of RrdDb.createFetchRequest() method when this method is called internally by this DataProcessor.
        Returns:
        Desired archive step (fetch resolution) in seconds.

      • setFetchRequestResolution

        public void setFetchRequestResolution​(long fetchRequestResolution)
        Sets desired RRD archive step in seconds to be used internally while fetching data from RRD files. In other words, this value will used as the last parameter of RrdDb.createFetchRequest() method when this method is called internally by this DataProcessor. If this method is never called, fetch request resolution defaults to 1 (smallest possible archive step will be chosen automatically).
        Parameters:
        fetchRequestResolution - Desired archive step (fetch resolution) in seconds.

      • setTimeZone

        public void setTimeZone​(TimeZone tz)
        Description copied from interface: DataHolder
        Set the time zone used for the legend.
        Specified by:
        setTimeZone in interface DataHolder
        Parameters:
        tz - the time zone to set

      • getEndingTimestamp

        public long getEndingTimestamp()
        Deprecated.
        Uses getEndTime() instead.
        Returns ending timestamp. Basically, this value is equal to the ending timestamp specified in the constructor. However, if the ending timestamps was zero, it will be replaced with the real timestamp when the processData() method returns. The real value will be calculated from the last update times of processed RRD files.
        Returns:
        Ending timestamp in seconds

      • getTimestamps

        public long[] getTimestamps()
        Returns consolidated timestamps created with the processData() method.
        Returns:
        array of timestamps in seconds

      • getValues

        public double[] getValues​(String sourceName)
        Returns calculated values for a single datasource. Corresponding timestamps can be obtained from the getTimestamps() method.
        Parameters:
        sourceName - Datasource name
        Returns:
        an array of datasource values
        Throws:
        IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

      • getAggregate

        @Deprecated
public double getAggregate​(String sourceName,
                           ConsolFun consolFun)
        Deprecated.
        Use Variable based method instead.
        Returns single aggregated value for a single datasource.
        Parameters:
        sourceName - Datasource name
        consolFun - Consolidation function to be applied to fetched datasource values. Valid consolidation functions are MIN, MAX, LAST, FIRST, AVERAGE and TOTAL (these string constants are conveniently defined in the ConsolFun class)
        Returns:
        a aggregate value as a double calculated from the source.
        Throws:
        IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

      • getAggregates

        @Deprecated
public Aggregates getAggregates​(String sourceName)
        Deprecated.
        Use Variable based method instead.
        Returns all (MIN, MAX, LAST, FIRST, AVERAGE and TOTAL) aggregated values for a single datasource.
        Parameters:
        sourceName - Datasource name
        Returns:
        Object containing all aggregated values
        Throws:
        IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

      • getVariable

        public Variable.Value getVariable​(String sourceName)
        Extract the variable value from an already define Variable datasource (a VDEF)
        Parameters:
        sourceName - Datasource name
        Returns:
        A combined time and value extracted calculated from the datasource

      • getVariable

        public Variable.Value getVariable​(String sourceName,
                                  Variable var)
        Returns single aggregated value for a single datasource.
        Parameters:
        sourceName - Datasource name
        var - variable that will generate value
        Returns:
        A combined time and value extracted calculated from the datasource

      • get95Percentile

        @Deprecated
public double get95Percentile​(String sourceName)
        Deprecated.
        Use Variable based method instead.
        This method is just an alias for getPercentile(String) method. Used by ISPs which charge for bandwidth utilization on a "95th percentile" basis.

        The 95th percentile is the highest source value left when the top 5% of a numerically sorted set of source data is discarded. It is used as a measure of the peak value used when one discounts a fair amount for transitory spikes. This makes it markedly different from the average.

        Read more about this topic at Rednet or Bytemark.

        Parameters:
        sourceName - Datasource name
        Returns:
        95th percentile of fetched source values

      • getPercentile

        @Deprecated
public double getPercentile​(String sourceName)
        Deprecated.
        Use Variable based method instead.
        Used by ISPs which charge for bandwidth utilization on a "95th percentile" basis.

        The 95th percentile is the highest source value left when the top 5% of a numerically sorted set of source data is discarded. It is used as a measure of the peak value used when one discounts a fair amount for transitory spikes. This makes it markedly different from the average.

        Read more about this topic at Rednet or Bytemark.

        Parameters:
        sourceName - Datasource name
        Returns:
        95th percentile of fetched source values

      • getPercentile

        @Deprecated
public double getPercentile​(String sourceName,
                            double percentile)
        Deprecated.
        Use Variable based method instead.
        The same as getPercentile(String) but with a possibility to define custom percentile boundary (different from 95).
        Parameters:
        sourceName - Datasource name.
        percentile - Boundary percentile. Value of 95 (%) is suitable in most cases, but you are free to provide your own percentile boundary between zero and 100.
        Returns:
        Requested percentile of fetched source values

      • getSourceNames

        public String[] getSourceNames()
        Returns array of datasource names defined in this DataProcessor.
        Returns:
        array of datasource names

      • getValues

        public double[][] getValues()
        Returns an array of all datasource values for all datasources. Each row in this two-dimensional array represents an array of calculated values for a single datasource. The order of rows is the same as the order in which datasources were added to this DataProcessor object.
        Returns:
        All datasource values for all datasources. The first index is the index of the datasource, the second index is the index of the datasource value. The number of datasource values is equal to the number of timestamps returned with getTimestamps() method.
        Throws:
        IllegalArgumentException - Thrown if invalid datasource name is specified, or if datasource values are not yet calculated (method processData() was not called)

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          Plottable plottable)
        Deprecated.
        Uses datasource(String, IPlottable) instead
        Adds a custom, plottable datasource (PDEF). The datapoints should be made available by a class extending Plottable class.
        Parameters:
        name - source name.
        plottable - class that extends Plottable class and is suited for graphing.

      • datasource

        public void datasource​(String name,
                       IPlottable plottable)
        Adds a custom, plottable datasource (PDEF). The datapoints should be made available by a class extending Plottable class.
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - source name.
        plottable - class that extends Plottable class and is suited for graphing.
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String rpnExpression)
        Deprecated.
        Uses datasource(String, String) instead

        Adds complex source (CDEF). Complex sources are evaluated using the supplied RPN expression.

        Complex source name can be used:
        • To specify sources for line, area and stack plots.
        • To define other complex sources.
        The supported RPN functions, operators and constants are detailed at RRD4J's wiki.

        Rrd4j does not force you to specify at least one simple source name as RRDTool.

        For more details on RPN see RRDTool's rrdgraph man page.

        Parameters:
        name - source name.
        rpnExpression - RPN expression containing comma delimited simple and complex source names, RPN constants, functions and operators.

      • datasource

        public void datasource​(String name,
                       String rpnExpression)

        Adds complex source (CDEF). Complex sources are evaluated using the supplied RPN expression.

        Complex source name can be used:
        • To specify sources for line, area and stack plots.
        • To define other complex sources.
        The supported RPN functions, operators and constants are detailed at RRD4J's wiki.

        Rrd4j does not force you to specify at least one simple source name as RRDTool.

        For more details on RPN see RRDTool's rrdgraph man page.

        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - source name.
        rpnExpression - RPN expression containing comma delimited simple and complex source names, RPN constants, functions and operators.
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String defName,
                          ConsolFun consolFun)
        Deprecated.
        Use Variable based method instead.
        Adds static source (SDEF). Static sources are the result of a consolidation function applied to any other source that has been defined previously.
        Parameters:
        name - source name.
        defName - Name of the datasource to calculate the value from.
        consolFun - Consolidation function to use for value calculation

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String sourceName,
                          double percentile)
        Deprecated.
        Use Variable based method instead.
        Creates a datasource that performs a percentile calculation on an another named datasource to yield a single value. Requires that the other datasource has already been defined; otherwise, it'll end up with no data
        Parameters:
        name - - the new virtual datasource name
        sourceName - - the datasource from which to extract the percentile. Must be a previously defined virtual datasource
        percentile - - the percentile to extract from the source datasource

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String defName,
                          Variable var)
        Deprecated.
        Uses datasource(String, String, Variable) instead.
        Creates a datasource that performs a variable calculation on an another named datasource to yield a single combined timestamp/value. Requires that the other datasource has already been defined; otherwise, it'll end up with no data
        Parameters:
        name - - the new virtual datasource name
        defName - - the datasource from which to extract the percentile. Must be a previously defined virtual datasource
        var - - a new instance of a Variable used to do the calculation

      • datasource

        public void datasource​(String name,
                       String defName,
                       Variable var)
        Creates a datasource that performs a variable calculation on an another named datasource to yield a single combined timestamp/value. Requires that the other datasource has already been defined; otherwise, it'll end up with no data
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - - the new virtual datasource name
        defName - - the datasource from which to extract the percentile. Must be a previously defined virtual datasource
        var - - a new instance of a Variable used to do the calculation
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String file,
                          String dsName,
                          ConsolFun consolFunc)
        Deprecated.
        Uses datasource(String, String, String, ConsolFun) instead.

        Adds simple datasource (DEF). Simple source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Parameters:
        name - source name.
        file - Path to RRD file.
        dsName - Datasource name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD

      • datasource

        public void datasource​(String name,
                       String file,
                       String dsName,
                       ConsolFun consolFunc)

        Adds simple datasource (DEF). Simple source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - source name.
        file - Path to RRD file.
        dsName - Datasource name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD
        Since:
        3.7

      • datasource

        public void datasource​(String name,
                       URI rrdUri,
                       String dsName,
                       ConsolFun consolFunc)

        Adds simple datasource (DEF). Simple source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - source name.
        rrdUri - URI to RRD file.
        dsName - Datasource name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String file,
                          String dsName,
                          ConsolFun consolFunc,
                          String backend)
        Deprecated.
        uses datasource(String, String, String, ConsolFun, RrdBackendFactory) instead

        Adds simple source (DEF). Source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Parameters:
        name - Source name.
        file - Path to RRD file.
        dsName - Data source name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
        backend - Name of the RrdBackendFactory that should be used for this RrdDb.

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String file,
                          String dsName,
                          ConsolFun consolFunc,
                          RrdBackendFactory backend)
        Deprecated.
        uses datasource(String, String, String, ConsolFun, RrdBackendFactory) instead

        Adds simple source (DEF). Source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Parameters:
        name - Source name.
        file - Path to RRD file.
        dsName - Data source name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
        backend - Name of the RrdBackendFactory that should be used for this RrdDb.

      • datasource

        public void datasource​(String name,
                       String file,
                       String dsName,
                       ConsolFun consolFunc,
                       RrdBackendFactory backend)

        Adds simple source (DEF). Source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - Source name.
        file - Path to RRD file.
        dsName - Data source name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
        backend - Name of the RrdBackendFactory that should be used for this RrdDb.
        Since:
        3.7

      • datasource

        public void datasource​(String name,
                       URI uri,
                       String dsName,
                       ConsolFun consolFunc,
                       RrdBackendFactory backend)

        Adds simple source (DEF). Source name can be used:

        • To specify sources for line, area and stack plots.
        • To define complex sources
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - Source name.
        uri - URI to RRD file.
        dsName - Data source name defined in the RRD file.
        consolFunc - Consolidation function that will be used to extract data from the RRD file ("AVERAGE", "MIN", "MAX" or "LAST" - these string constants are conveniently defined in the ConsolFun class).
        backend - Name of the RrdBackendFactory that should be used for this RrdDb.
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          FetchData fetchData)
        Deprecated.
        Uses datasource(String, FetchData) instead.
        Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it.
        Parameters:
        name - Source name.
        fetchData - Fetched data containing values for the given source name.

      • datasource

        public void datasource​(String name,
                       FetchData fetchData)
        Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it.
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - Source name.
        fetchData - Fetched data containing values for the given source name.
        Since:
        3.7

      • addDatasource

        @Deprecated
public void addDatasource​(String name,
                          String dsName,
                          FetchData fetchData)
        Deprecated.
        Uses datasource(String, String, FetchData) instead.
        Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it. The values will be extracted from dsName in fetchData.
        Parameters:
        name - Source name.
        dsName - Source name in the fetch data.
        fetchData - Fetched data containing values for the given source name.

      • datasource

        public void datasource​(String name,
                       String dsName,
                       FetchData fetchData)
        Adds DEF datasource with datasource values already available in the FetchData object. This method is used internally by Rrd4j and probably has no purpose outside of it. The values will be extracted from dsName in fetchData.
        Specified by:
        datasource in interface DataHolder
        Parameters:
        name - Source name.
        dsName - Source name in the fetch data.
        fetchData - Fetched data containing values for the given source name.
        Since:
        3.7

      • processData

        public void processData()
                 throws IOException
        Method that should be called once all datasources are defined. Data will be fetched from RRD files, RPN expressions will be calculated, etc.
        Throws:
        IOException - Thrown in case of I/O error (while fetching data from RRD files)

      • getValuesPerPixel

        public double[] getValuesPerPixel​(String sourceName,
                                  int pixelCount)
        Method used to calculate datasource values which should be presented on the graph based on the desired graph width. Each value returned represents a single pixel on the graph. Corresponding timestamp can be found in the array returned from getTimestampsPerPixel() method.
        Parameters:
        sourceName - Datasource name
        pixelCount - Graph width
        Returns:
        Per-pixel datasource values
        Throws:
        IllegalArgumentException - Thrown if datasource values are not yet calculated (method processData() was not called)

      • getValuesPerPixel

        public double[] getValuesPerPixel​(String sourceName)
        Method used to calculate datasource values which should be presented on the graph based on the graph width set with a setPixelCount(int) method call. Each value returned represents a single pixel on the graph. Corresponding timestamp can be found in the array returned from getTimestampsPerPixel() method.
        Parameters:
        sourceName - Datasource name
        Returns:
        Per-pixel datasource values
        Throws:
        IllegalArgumentException - Thrown if datasource values are not yet calculated (method processData() was not called)

      • getTimestampsPerPixel

        public long[] getTimestampsPerPixel​(int pixelCount)
        Calculates timestamps which correspond to individual pixels on the graph. It also set the timestampsPerPixel value.
        Parameters:
        pixelCount - Graph width
        Returns:
        Array of timestamps

      • getTimestampsPerPixel

        public long[] getTimestampsPerPixel()
        Calculates timestamps which correspond to individual pixels on the graph based on the graph width set with a setPixelCount(int) method call.
        Returns:
        Array of timestamps

      • dump

        public String dump()
        Dumps timestamps and values of all datasources in a tabular form. Very useful for debugging.
        Returns:
        Dumped object content.

      • getLastRrdArchiveUpdateTime

        public long getLastRrdArchiveUpdateTime()
        Returns time when last RRD archive was updated (all RRD files are considered).
        Returns:
        Last archive update time for all RRD files in this DataProcessor

      • setEndTime

        public void setEndTime​(long time)
        Description copied from interface: DataHolder
        Sets the time when the graph should end. Time in seconds since epoch (1970-01-01) is required. Negative numbers are relative to the current time.
        Specified by:
        setEndTime in interface DataHolder
        Parameters:
        time - Ending time for the graph in seconds since epoch
        Since:
        3.7

      • getEndTime

        public long getEndTime()
        Description copied from interface: DataHolder
        Returns ending timestamp.
        Specified by:
        getEndTime in interface DataHolder
        Returns:
        Ending timestamp in seconds
        Since:
        3.7

      • setStartTime

        public void setStartTime​(long time)
        Description copied from interface: DataHolder
        Sets the time when the graph should start. Time in seconds since epoch (1970-01-01) is required. Negative numbers are relative to the current time.
        Specified by:
        setStartTime in interface DataHolder
        Parameters:
        time - Starting time for the graph in seconds since epoch

      • getStartTime

        public long getStartTime()
        Description copied from interface: DataHolder
        Returns starting timestamp.
        Specified by:
        getStartTime in interface DataHolder
        Returns:
        Starting timestamp in seconds
        Since:
        3.7

      • setTimeSpan

        public void setTimeSpan​(long startTime,
                        long endTime)
        Description copied from interface: DataHolder
        Sets starting and ending time for the for the graph. Timestamps in seconds since epoch are required. Negative numbers are relative to the current time.
        Specified by:
        setTimeSpan in interface DataHolder
        Parameters:
        startTime - Starting time in seconds since epoch
        endTime - Ending time in seconds since epoch
        Since:
        3.7