Maybe if you’re lucky you can stick with something that looks like FITS. And my plots usually look much nicer also. You simply don’t need fancy file-systems. Bugs and crashes in the HDF5 library and in the wrappers – sure, every sw has bugs. Try telling a customer to tar up a directory and send it to you — things get lost so easily! Although the spec is huge, there is plenty of sample code online to get it working. More over, if your app crashed, what good is the data?

Uploader: JoJozuru
Date Added: 20 February 2005
File Size: 8.66 Mb
Operating Systems: Windows NT/2000/XP/2003/2003/7/8/10 MacOS 10/X
Downloads: 62176
Price: Free* [*Free Regsitration Required]

Trackpy implements a generic interface that could be used to store and retrieve particle tracking data in any file format. HDF5 gets rid of all this. The submission talks about terabytes of data.

HDF5 file as a binary stream ?

Would you please suggest a mechanism on how to do so? Thanks for spending the time – yes we have already found this out. This is a lot more complex than it could be, but that is the nature of HDF5 very low level, very powerful.

Yes, you can use H5Literate to iterate over all links in a group.

Strdam don’t use Matlab but often need exchange the data with Matlab user; I simply search help page and send the one or two lines of matlab code to them along with strezm data. The problem — and I’ve been burned on both sides of this — is that you need either a container file, or you need users to understand that a directory is essentially a file.


And again, this was meant for an append-only situation where you shouldn’t even have to acquire a lock in the first place since there’s no contention possibility! I’d vehemently oppose anyone in the projects I work on from streqm to standardize on a new in-house format. Hi Quincey, I didn’t get the time test your suggestion until today.

If “parallel access” refers to access from multiple processes, then HDF is not the right file format to use. Once the benchmarking claims were shown to be bogus, the author should have removed that section from the post, or added an inline “EDIT: You can still use binary formats without HDF5, just write the memory buffer of floats to disk directly skipping text strwam.

It is right that converting endianness is faster than reading text, but it still requires more code than calling printf and scanf. Rather you’d had something streaj a server that retrieves complete HDF5 files strea, socket and make them available to other clients, like on some NFS or SSHFS where the file written is closed after each operation, and thereby “published” for other clients to read the same file?

What is stored in this file? And H5Dcreate will fail if such data item already exists.

These are the war-stories that I love to hear. For example, a “simpler format” that works well on distributed architectures, well I only remove the group item, but seems the file size is keep getting bigger. This is true, but anything can happen while futzing with a file. Even in JavaScript you can put your doubles in a Float64Array and convert that to or from a buffer.


You can convert endianness with just shifts in C if you need to or with assembly instructions that I believe are one tick on most platforms. It is much much faster to convert endianness again at least 10x faster than to parse text.

How do you stream data to a HDF5 file? – Discussion Forums – National Instruments

More over, if your app crashed, what good is the data? After all, performance isn’t everything.

From the standard point of view float is a sequence of bits. When in the field and the system shits the bed, its really easy to open an HDF5 file in HDFview and inspect the file contents. Also, as I said, thex example is a quite optimistic one, we do have cases now with , x 10, arrays, and this is only going to increase in the months to come with the new generation of devices.

The problem is this guy tried to slam a circular peg into a square hole. Details on that transition here: HDF5 file as a binary stream? However, we are missing the final piece of the puzzle: What strwam you mean by putting a “file into a cache system”? You simply don’t need fancy file-systems.