Helioclim-3 automatic access

Last update: May 2017

    Build your first WGET request in 5 minutes - WGET - CURL - Error messages - FAQ

The automatic access to HelioClim-3 data is well-suited to automatize the retrieve of data, for instance for monitoring applications. Download WGET.EXE


 

1_ Build your first HC3 WGET request in 5 minutes

  1. Subscribe an annual access to HelioClim-3 (Contact us)
  2. Once you receive your login and password, build your first WGET request (permanent free automatic access to Carpentras, France (44.083°, 5.059°) using the login/password "guest/guest", from 2017-01-01 to 2017-01-05, 15 min time step, plane oriented 15° south, takes into account the shadowing effect due to the far horizon, return a code of error - Shift to the mirror server if no answer):
  3. Launch your command, and retrieve the hc3v5_Carpentras.csv
    • WINDOWS: double click on the .bat file
    • UNIX: launch the command .sh
  4. For more details on Wget and Curl commands, on errors, on more advanced commands, please read following sections and FAQ

2_ Build wget requests

Request (Unix and Windows):
wget -O output.csv --header="soda-user: login" --header=soda-passwd: password" ...
"http://server/pub/hc3v5_request.php?
geopoint=latitude,longitude&elevation=altitude& ...
firstday=yyyy-mm-dd&lastday=yyyy-mm-dd&duration=time_step&time=time_reference& ...
slope=tilt_of_panels&azimuth=az_of_panels&albedo=ground_albedo&horizon=boolean_horizon& ...
header=boolean_header&rely=boolean_code&format=hc3_format"

With (inputs):

  • login/password authentication: login and password provided by the SoDa team to access HelioClim-3 within the framework of your subscription. Note that a free permanent automatic access is set up for the site of Carpentras, France, where a Baseline Surface Radiation Network (BSRN) station is available with the login/password guest/guest
  • server: "www.soda-is.com" (main server) or "pro.soda-is.com" (backup-mirror server)
  • latitude and longitude: in degrees, with at least 3 digits after comma
  • altitude: in meters. Set "elevation=-999" to let SoDa get the elevation from Nasa SRTM database
  • yyyy-mm-dd: set the first and the last days of the request
  • time_step: "15" (minutes), "h" (hourly), "d" (daily), "m" (monthly)
  • time_reference: "TU" (universal time) or "TST" (True Solar Time)
  • tilt_of_panels: from "0" (horizontal) to "90", and "-999" (2D tracking)
  • az_of_panels: from "0" (North), 90° (East), 180° (South) to 270° (West), and "-999" (tracking)
  • ground_albedo: from "0.0" to "1.0". A usual inland value is "0.2"
  • boolean_horizon: optional. Takes into account "1" or not "0" the far horizon. If not set, default value is "0"
  • boolean_header: optional. Return ("1") or not ("0") the header of the file. If not set, default value is "0" (no header)
  • boolean_code: optional. Return ("1") or not ("0") a code of error (0: no data, 1: sun below horizon, 2: satellite estimation=OK, 5: interpolation, 6: forecasts). If not set, the code column is not provided
  • hc3_format: optional. Unique possible value is "unified". If not set, default is the usual standard format. Download an example of this new output format for a request over the horizontal plane, and for a fix-tilted or normal plane.

Example: wget -O SoDa_HC3v5_Carpentras_2017-01-01_to_2017-01-05_tilt15_az180_horizon1.csv --header="soda-user: guest" --header=soda-passwd: guest" ...
"http://www.soda-is.com/pub/hc3v5_request.php?geopoint=44.083,5.059&elevation=-999& ...
firstday=2017-01-01&lastday=2017-01-05&duration=15&time=TU&slope=15&azimuth=180&albedo=0.2"

Outputs: identical to those retrieved from the SoDa website, except that the separator is the space instead of the semi-colon. The time period max retrieved using the wget command is: one month for a 1 to 5 min time step, one year for a 10 min up to one hour time step, and unlimited for the other time steps.

Warning! It is strongly recommanded to avoid the launch of parallel requests, using the "&" at the end of each wget request line. This would endanger our system.

Download examples: (all examples include an automatic shift to the backup server) (Download wget.exe only)

Example WINDOWS UNIX
Simple, HC3v5 on 2017-01-01 to 2017-01-05 download example download example
Get HC3v4 for yesterday download example download example
Get HC3v5 for the previous full month download example download example

Top of page

3_Build CURL requests

Request (Unix and Windows):

curl -F "stations=@stations.txt" -o output.csv --header "soda-user: login" ...
--header "soda-passwd:
password"
http://server/pub/hc3v5_request.php

Where the "stations.txt" file gathers the information of the sites as follows:
#id lat lon elevation UT time_step first_date last_date slope azimuth albedo [horizon]

Carpentras1 44.083 5.059 -999 UT 15 2012-01-01 2012-12-31 15 180 0.2 1
Carpentras2 44.083 5.059 -999 UT 15 2012-01-01 2012-12-31 30 180 0.2 1
...
if [horizon] is not set, the default value is "0" (the shadowing effect of the far horizon is not taken into account). Please note that the parameters "header" and "rely" defined in Wget section are not available with the CURL command.

Download examples: (all examples include an automatic shift to the backup server)

Example WINDOWS UNIX
Simple, HC3v5 on 2017-01-01 to 2017-01-05 download example download example
Get HC3v4 for yesterday download example download example
Get HC3v5 for the previous full month download example download example

Top of page

4_ Error messages

  • ERROR -1: Incorrect tilted plane parameter
  • ERROR -2: Cannot open file
  • ERROR -3: Cannot write to file
  • ERROR -4: Cannot read file
  • ERROR -5: Not enough memory
  • ERROR -6: Cannot access remote data
  • ERROR -7: Helioclim service is corrupted
  • ERROR -8: Unregistered IP address
  • ERROR -9: Incorrect location parameter
  • ERROR -10: Incorrect time step parameter
  • ERROR -11: Incorrect date parameter
  • ERROR -12: Incorrect horizon parameter
  • ERROR -13: Incorrect elevation parameter
  • ERROR -14: Incorrect stations file
  • ERROR -15: Subprogram failed with no error message
  • ERROR -16: Incorrect calibration parameter
  • ERROR -17: Incorrect user/password parameters
  • ERROR -18: Incorrect albedo parameter
  • ERROR -19: Incorrect time reference parameter
  • ERROR -21: Incorrect rely parameter
  • ERROR -22: Incorrect header parameter
  • ERROR -23: HC3 database is not updated
  • ERROR -24: This service is out the scope of your SoDa subscription
  • ERROR -25: Incorrect gamma-sun-min parameter
  • ERROR -26: Mismatch dimensions of returned parameters
  • ERROR 1: No data available with these input parameters
    (generic message if the error is unknown)

Top of page

5_ FAQ

I_ "I launched this example script, but I can't retrieve the output file!"

In this case, you probably receive a "time out message".

  1. Network shortage in OUR side: to check this point, try the mirror SoDa server ("http://www.soda-is.com/" or "http://pro.soda-is.com/").
  2. Network shortage on YOUR side: just try to go on the internet.
  3. Problem of proxy: your internet browser knows which "door" (proxy, firewall...) to use to get out of your local network, but you need to tell wget the information relative to this proxy.
    For wget, you need to specify the proxy information in your .bat file by adding:
    set http_proxy=machine:port
    before the wget command. 1-2-3... launch!
    The proxy information (machine and port) are available:
        Internet Explorer: Tools / Options / Connections / Network parameters
        Firefox: Tools / Options / Advanced / Network / Parameters
    Alternatively, you may ask for the proxy information to your IT person.
  II_ "Why two requests inside the same pixel MSG return 2 different irradiation values?"

The HelioClim databases consist of Global Irradiance values over the horizontal plane. These values take into account the default elevation value from the SRTM database (if "elevation=-999"). Two requests made in the same pixel return different results due to a correction with respect to the elevation.

  III_ "How can I change the spaces by semi colons in my output file?"                              

Solution 1 "manual": in the top menu "Data" in Excel (Microsoft Office Excel 2007), select the sub-menu "convert". Select the "space" character and Excel will automatically display one figure per column.

Solution 2 "automatic":
In Windows: click here to download sed.exe + libraries. To replace automatically the " " by ";" in your test.txt file obtained using a wget command, you can use the following command:
sed "s/ /;/g" test.txt > test.csv
As a consequence, you can create a .bat file with the following command lines: wget -O test.txt "http://www.soda-is.com/pub/hc3v5_request.php?geopoint=44.083,5.059&elevation=-999& ...
firstday=2017-01-01&lastday=2017-01-05&duration=15&time=TU&slope=15&azimuth=180&albedo=0.2"
sed "s/ /;/g" test.txt > test.csv
del test.txt

=> the file test.txt will be automatically deleted, and replaced by the test.csv file with ";" as separator.

In Unix, sed is a default tool, so you just have to copy and paste the previous example in a .sh file, by taking care of beginning the file by #!/bin/sh.

  IV_ "How to launch automatically my requests at night?";
In Unix, launch automatically your script with a "cron". In Windows, use the "scheduled tasks" to launch it automatically in a Windows environment.

Top of page

  V_ "How can I introduce a delay between two successive wget requests?"               

In Unix: sleep [N]

In Windows: timeout [N]

...where "N" is the number of seconds to wait.

Top of page

 VI_ "Why doesn't SoDa do the switch to the mirror website by itself? "
A few SoDa customers asked why we didn't opt for a solution could be based on DNS or IP Failover. The reason is that this type of solution doesn't work when the two servers are geographically far away from each other.

To handle this case, a third machine is required which would receive all the requests, and would send them to the two servers depending on their availability. This would lose the advantage of having two synchronized servers on which you can switch from one to the other in case of failure.

So far the only solution is that you code by yourself this switch, as given in the examples available for download on this page.
 

Top of page