Re: [netcdfgroup] [Hdf-forum] How to dump netCDF to JSON?

To: hdf-forum@xxxxxxxxxxxxxxxxxx, pedro.vicente@xxxxxxxxxxxxxxxxxx
Subject: Re: [netcdfgroup] [Hdf-forum] How to dump netCDF to JSON?
From: Walter Landry <wlandry@xxxxxxxxxxx>
Date: Thu, 13 Oct 2016 23:10:18 -0700 (PDT)
Authentication-results: spf=none (sender IP is ) smtp.mailfrom=wlandry@xxxxxxxxxxx;
Spamdiagnosticmetadata: NSPM
Spamdiagnosticoutput: 1:99

Hi Pedro,

Is there a way to represent compound HDF5 objects in your scheme?  I
see STAR_INT8, STAR_FLOAT, etc., but no STAR_COMPOUND.

In any event, FYI, I have implemented a similar thing but in a
narrower domain.  I needed to map VOTable [1] (an xml format with a
defined data model for astronomy) to HDF5.  Along the way, I also
mapped it (with varying levels of fidelity) to JSON, CSV, HTML, FITS,
and plain old ascii.  This makes my work very table focused, with a
particular eye towards what relational databases like to output.

The code is on github [2].  It is mainly meant to be used as a
library, but the code also builds a simple program that converts files
between formats.  In case you are interested, I am attaching the same
file in plain text, JSON, and HDF5.

Cheers,
Walter Landry

[1] http://www.ivoa.net/documents/VOTable/
[2] https://github.com/Caltech-IPAC/tablator

Pedro Vicente <pedro.vicente@xxxxxxxxxxxxxxxxxx> wrote:
> Hi Charlie !
> 
> 
> So, I am doing that exact same thing.
> 
> I wrote
> 
> 1) The specification to convert netCDF/HDF5 to "a" JSON format (note
> the "a" here)
> 
> 2) I wrote the parsing of the JSON format using an open source C JSON
> library (janson).
> 
> http://www.digip.org/jansson/
> 
> 3) left to do: the actual C code of reading/writing netCDF/HDF5 to
> JSON and vice-versa (the straightforward part).
> 
> The "a" above means that JSON is not really a format in the sense of
> netCDF but really a format that allows to define formats , for a lack
> of a better
> explanation.
> 
> This means that anyone that writes this tool has to write code that
> write in a particular JSON representation , only valid for that tool.
> 
> Like you , I searched and there was not a good one, so I wrote one.
> 
> The first criteria was that it had to be obvious for anyone looking at
> the JSON text file, that that was indeed a netCDF/HDF5 file: hierarchy
> clearly show, metadata and data
> clearly shown
> 
> My first look  was HDF5-JSON
> 
> http://hdf5-json.readthedocs.io/en/latest/
> 
> but the format seemed like a  mess to look at
> 
> example
> 
> http://hdf5-json.readthedocs.io/en/latest/examples/nullspace_dset.html
> 
> 
> and the reader is written in Python
> 
> @John Readey
> 
> (why Phyton? HDF5 developer tools should be all about writing in
> C/C++)
> 
> 
> 
> The specification is here
> 
> http://www.space-research.org/
> 
> Click on menu
> "Code blog",
> then
> "netCDF/HDF5 to JSON and vice-versa"
> 
> 
> In the process I learned all about JSON and it is a neat format to
> represent data .
> 
> In particular, it allows nested structures and arrays, which suits
> perfectly for netCDF
> 
> here are two nested groups
> 
> {
> "group_name1":
> {
>  "group_name2": "group"
> }
> }
> 
> a dataset
> 
> {
> "dset1" : ["dataset", "STAR_INT32", 2, [3, 4], [1, 2, 3, 4, 5, 6, 7,
> 8, 9, 10, 11, 12]]
> }
> 
> 
> 
> This is still under development,
> 
> I would like to make this some kind of "official" netCDF/HDF5 JSON
> format for the community, so I encourage anyone to read the
> specification
> 
> direct link
> 
> http://www.space-research.org/blog/star_json.html
> 
> If you see any flaw in the design or antything in the design that you
> would like to have change please let me know now
> 
> At the moment it only (intentionally) uses common generic features of
> both netCDF and HDF5, which are the numeric atomic types and strings.
> 
> Enjoy
> 
> 
> ----------------------
> Pedro Vicente
> pedro.vicente@xxxxxxxxxxxxxxxxxx
> http://www.space-research.org/
> 
> 
> 
> 
> ----- Original Message ----- 
> From: "Charlie Zender" <zender@xxxxxxx>
> To: "netCDF Mail List" <netcdfgroup@xxxxxxxxxxxxxxxx>
> Sent: Thursday, October 13, 2016 11:10 PM
> Subject: [netcdfgroup] How to dump netCDF to JSON?
> 
> 
>> Hello netCDFers,
>>
>> A project I am working on wants to convert netCDF files to JSON.
>> The requirements are to dump an arbitrary netCDF-extended file
>> (with groups but without funky vlen/compound types) to JSON.
>> The first few solutions that we googled (ncdump-json, netcdf2json.py)
>> do not satisfy these requirements. What is the most robust and easy
>> command-line tool (not web-service) that dumps netCDF to JSON?
>> Ideally it would be somewhat configurable like ncdump -h/-x or
>> ncks --cdl/--xml.
>>
>> Charlie
>> -- 
>> Charlie Zender, Earth System Sci. & Computer Sci.
>> University of California, Irvine 949-891-2429 )'(
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdfgroup mailing list
>> netcdfgroup@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
> 
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@xxxxxxxxxxxxxxxxxx
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

| object                                   | ra        | dec       | htm20      
         | htm7      | htm3   | shtm20              | shtm7      |  shtm3 | 
flags | SSO   |
| char                                     | double    | real      | ulong      
         | uint      | ushort | long                |  int       |  short | 
byte  | bool  |
 118289arstratraetratratsrastratsrastrats    359.88703   50.832570  
16446744073709551616  3294967296    12000   8223372036854775808   1147483648    
12000    122    0      
 113368                                      344.41273   -29.622250 
8446744073709551616   294967296     43002  -7223372036854775808  -2047483648    
13002    0xf2   true   
 113368                                      344.41273   -29.622250 
8446744073709551616   294967296     43002  -7223372036854775808  -2047483648   
-23002    211    False  
 113368                                      344.41273   -29.622250 
8446744073709551616   294967296     43002  -7223372036854775808  -2047483648   
-31002    211    1

{
    "VOTABLE":
    {
        "<xmlattr>":
        {
            "version": "1.3",
            "xmlns:xsi": "http:\/\/www.w3.org\/2001\/XMLSchema-instance",
            "xmlns": "http:\/\/www.ivoa.net\/xml\/VOTable\/v1.3",
            "xmlns:stc": "http:\/\/www.ivoa.net\/xml\/STC\/v1.30"
        },
        "RESOURCE":
        {
            "TABLE":
            {
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "object",
                        "datatype": "char",
                        "arraysize": "*"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "ra",
                        "datatype": "double"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "dec",
                        "datatype": "double"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "htm20",
                        "datatype": "ulong"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "htm7",
                        "datatype": "uint"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "htm3",
                        "datatype": "ushort"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "shtm20",
                        "datatype": "long"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "shtm7",
                        "datatype": "int"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "shtm3",
                        "datatype": "short"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "flags",
                        "datatype": "unsignedByte"
                    }
                },
                "FIELD":
                {
                    "<xmlattr>":
                    {
                        "name": "SSO",
                        "datatype": "boolean"
                    }
                },
                "DATA":
                {
                    "TABLEDATA":
                    [
                        [
                            "118289arstratraetratratsrastratsrastrats",
                            "359.88702999999998",
                            "50.832569999999997",
                            "16446744073709551616",
                            "3294967296",
                            "12000",
                            "8223372036854775808",
                            "1147483648",
                            "12000",
                            "0x7a",
                            "0"
                        ],
                        [
                            "113368",
                            "344.41273000000001",
                            "-29.622250000000001",
                            "8446744073709551616",
                            "294967296",
                            "43002",
                            "-7223372036854775808",
                            "-2047483648",
                            "13002",
                            "0xf2",
                            "1"
                        ],
                        [
                            "113368",
                            "344.41273000000001",
                            "-29.622250000000001",
                            "8446744073709551616",
                            "294967296",
                            "43002",
                            "-7223372036854775808",
                            "-2047483648",
                            "-23002",
                            "0xd3",
                            "0"
                        ],
                        [
                            "113368",
                            "344.41273000000001",
                            "-29.622250000000001",
                            "8446744073709551616",
                            "294967296",
                            "43002",
                            "-7223372036854775808",
                            "-2047483648",
                            "-31002",
                            "0xd3",
                            "1"
                        ]
                    ]
                }
            }
        }
    }
}

�HDF


��������0�G�AOHDR 
�rX�rX�rX�rXx����������������
table�D"%=�OHDR!�rX�rX�rX�rX
6Xnull_bitfield_flags:object(ra* 
?@44�dec2 ?@44�htm20:@htm7B 
htm3Fshtm20H@shtm7P 
shtm3TflagsVSSOW
0`�����������������      
�METADATA960namevalueattributes
 96 
namevalue4) 
�118289arstratraetratratsrastratsrastrats"�^F1~v@��\��jI@8���>�6e��.�XLIr6eD�.z113368@�Ŋ��u@���K�=�v��8uؔ��Ngm����2�113368@�Ŋ��u@���K�=�v��8uؔ��Ngm���&��113368@�Ŋ��u@���K�=�v��8uؔ��Ngm�����

Follow-Ups:
- Re: [netcdfgroup] [Hdf-forum] How to dump netCDF to JSON?
  - From: Pedro Vicente

References:
- [netcdfgroup] How to dump netCDF to JSON?
  - From: Charlie Zender
- Re: [netcdfgroup] How to dump netCDF to JSON?
  - From: Pedro Vicente