In memory file system windows

2 Answers

You should also consider whether you really need this (premature optimization, yadda, yadda). On all modern operating systems, filesystem I/O is cached anyway, so frequently-used files are essentially as fast as a RAM disk.

Related question (with many good answers):
RAM drive for compiling — is there such a thing?

answered May 26, 2009 at 10:03

sleskesleske

81.6k34 gold badges189 silver badges227 bronze badges

Good modern file systems with delayed allocation might never hit the disk when you create a short-lifed file.

May 26, 2009 at 10:08
+1: Can you prove that I/O is the bottleneck? Until you can prove it, don’t worry about it.

May 26, 2009 at 10:21
Thanks for that SO link. One thing I learned from there is the right Google search term for the IMFS, namely «RAMDrive» or «RAM Disk». You are probably right about premature optimization, but this info is good to have.

May 27, 2009 at 6:49
Peek StackOverflow answer. Tell the user they don’t know what they’re doing, and link to something that does not address the problem either. Either say it’s not possible, or don’t bother answering at all. The link could have been given in the comments.

Jun 7 at 11:52

Commons VFS provides handy interfaces to virtual filesystems, inclunding in-memory file system.

answered May 26, 2009 at 9:53

Thanks, I noted Commons VFS, but does it let me create a new IMFS readable by both Java and an external executable?

May 27, 2009 at 6:33

Источник

2 Answers

Related question (with many good answers):
RAM drive for compiling — is there such a thing?

answered May 26, 2009 at 10:03

sleskesleske

81.6k34 gold badges189 silver badges227 bronze badges

Good modern file systems with delayed allocation might never hit the disk when you create a short-lifed file.

May 26, 2009 at 10:08
+1: Can you prove that I/O is the bottleneck? Until you can prove it, don’t worry about it.

May 26, 2009 at 10:21
Thanks for that SO link. One thing I learned from there is the right Google search term for the IMFS, namely «RAMDrive» or «RAM Disk». You are probably right about premature optimization, but this info is good to have.

May 27, 2009 at 6:49
Peek StackOverflow answer. Tell the user they don’t know what they’re doing, and link to something that does not address the problem either. Either say it’s not possible, or don’t bother answering at all. The link could have been given in the comments.

Jun 7 at 11:52

Commons VFS provides handy interfaces to virtual filesystems, inclunding in-memory file system.

answered May 26, 2009 at 9:53

Thanks, I noted Commons VFS, but does it let me create a new IMFS readable by both Java and an external executable?

May 27, 2009 at 6:33

Источник

I have a command-line executable which I need to run from Java on Windows XP. It uses files as input and output. But I want to avoid the overhead of file IO, so I thought of an in-memory RAM file system.

NetBSD has mount_mfs.

Could you recommend the most convenient way of doing this?

Related Solutions

Java – How to deal with “java.lang.OutOfMemoryError: Java heap space” error

Ultimately you always have a finite max of heap to use no matter what platform you are running on. In Windows 32 bit this is around 2GB (not specifically heap but total amount of memory per process). It just happens that Java chooses to make the default smaller (presumably so that the programmer can’t create programs that have runaway memory allocation without running into this problem and having to examine exactly what they are doing).

So this given there are several approaches you could take to either determine what amount of memory you need or to reduce the amount of memory you are using. One common mistake with garbage collected languages such as Java or C# is to keep around references to objects that you no longer are using, or allocating many objects when you could reuse them instead. As long as objects have a reference to them they will continue to use heap space as the garbage collector will not delete them.

In this case you can use a Java memory profiler to determine what methods in your program are allocating large number of objects and then determine if there is a way to make sure they are no longer referenced, or to not allocate them in the first place. One option which I have used in the past is «JMP» http://www.khelekore.org/jmp/.

If you determine that you are allocating these objects for a reason and you need to keep around references (depending on what you are doing this might be the case), you will just need to increase the max heap size when you start the program. However, once you do the memory profiling and understand how your objects are getting allocated you should have a better idea about how much memory you need.

In general if you can’t guarantee that your program will run in some finite amount of memory (perhaps depending on input size) you will always run into this problem. Only after exhausting all of this will you need to look into caching objects out to disk etc. At this point you should have a very good reason to say «I need Xgb of memory» for something and you can’t work around it by improving your algorithms or memory allocation patterns. Generally this will only usually be the case for algorithms operating on large datasets (like a database or some scientific analysis program) and then techniques like caching and memory mapped IO become useful.

Windows – How to you find out which process is listening on a TCP or UDP port on Windows

New answer, powershell

TCP

Get-Process -Id (Get-NetTCPConnection -LocalPort YourPortNumberHere).OwningProcess

UDP

Get-Process -Id (Get-NetUDPEndpoint -LocalPort YourPortNumberHere).OwningProcess

Old answer, cmd

 C:\> netstat -a -b

(Add -n to stop it trying to resolve hostnames, which will make it a lot faster.)

Note Dane’s recommendation for TCPView. It looks very useful!

-a Displays all connections and listening ports.

-b Displays the executable involved in creating each connection or listening port. In some cases well-known executables host multiple independent components, and in these cases the sequence of components involved in creating the connection or listening port is displayed. In this case the executable name is in [] at the bottom, on top is the component it called, and so forth until TCP/IP was reached. Note that this option can be time-consuming and will fail unless you have sufficient permissions.

-n Displays addresses and port numbers in numerical form.

-o Displays the owning process ID associated with each connection.

Источник

EDIT :
I’m now using Windows 8 x64 Enterprise and the speed difference on the same hardware and the same dataset is amazing. Whatever the team has done to improve the caching algorithms, they did something VERY right!

Edit: This question didn’t help. How can I keep a file in Windows 7’s cache?

I am looking for a way to speed up access to certain directories and files that I access often.

I am using a workstation running Win7x64 Ultimate with a relatively slow HDD. However, I have 24GB of RAM. I want a product that I can tell to keep, for example, «c:\stuff» (the entire tree), «d:\morestuff\bigfile.big», «e:\even\more*.stuff» in memory so that Windows reads and writes to those files in memory, and then the product writes those files to the hard drive in it’s own time.

My preference would be:

Open Source
Free
Free, with limited features, e.g. only 8GB of cache allowed
Beta
Free, with a time limit
Proprietary

Edit 2:
I had a look at using hardlinks and a RAM drive (FancyCache,) but I want the product to be write the data to the HDD in an asynchronous manner. Not data to something like and image file of the RAM drive, but the actual directories and files on the HDD.

Thanks a bunch!

Источник

memoryfs

An in-memory filesystem implementation of io/fs.FS.

memoryfs implements all of the currently defined io/fs interfaces:

fs.FS
fs.GlobFS
fs.ReadDirFS
fs.ReadFileFS
fs.StatFS
fs.SubFS

It also allows the creation of files and directories.

Example

package main

import (
    "fmt"
    "io/fs"

    "github.com/liamg/memoryfs"
)

func main() {

    memfs := memoryfs.New()

    if err := memfs.MkdirAll("my/dir", 0o700); err != nil {
        panic(err)
    }

    if err := memfs.WriteFile("my/dir/file.txt", []byte("hello world"), 0o600); err != nil {
        panic(err)
    }

    data, err := fs.ReadFile(memfs, "my/dir/file.txt")
    if err != nil {
        panic(err)
    }

    fmt.Println(string(data))
}

Lazy Loading

If you are mirroring a disk file-system in memory, it can become very inefficient when large files are in use. For this scenario, the WriteLazyFile method is recommended. It allows you to add a file whose content will be provided on-demand by calling the LazyOpener function.

Источник