Beefy Boxes and Bandwidth Generously Provided by pair Networks
Do you know where your variables are?
 
PerlMonks  

Re^3: Perl on Windows Best Practices?

by bulk88 (Priest)
on Jul 31, 2012 at 03:20 UTC ( #984555=note: print w/replies, xml ) Need Help??


in reply to Re^2: Perl on Windows Best Practices?
in thread Perl on Windows Best Practices?

SMB/My Network Places/CIFS handshaking/latency is slower than any HD IMHO or even HTTP. Look at it in wireshark one day, many handshake back and forth set up stuff, lots of metadata queries on the files, attribute reading and setting, file pointer setting, etc. Dozens or hundreds of sub 100KB files gets slow and will not saturate a network connection mbps wise in my experience between Windows computers. Of course copying files with Explorer and Perl interp starting up and loading PMs are both synchronous I/O. I tried copying my perl dir from 1 XP machine to a Server 2003 machine with Explorer, no HD light flashing on the source machine (the whole perl dir was in read cache from a recent grep of the dir), probably none on the Server 2003 machine (8 GB of RAM, 7 GB probably free), the average was 1-2 mbps up (to dest) and 500 kbps down (from dest) when copying .pl and .pm files, and jumped to 70 mbps and 500 kbps down (from dest) when copying .pdb files. I have a 100mbps ethernet network. I dont think file caching works over SMB connections, atleast with NT 5, since multiple clients or the server itself can read write the file at any time and SMB has no way of pushing cache invalidation messages across the wire or doesn't use them in most cases.

It seems things improved with NT 6, http://en.wikipedia.org/wiki/Server_Message_Block#SMB2. I have no NT 6 machines.

Replies are listed 'Best First'.
Re^4: Perl on Windows Best Practices?
by BrowserUk (Pope) on Jul 31, 2012 at 04:05 UTC
    I dont think file caching works over SMB connections,

    Hm. l: is my net use for live.sysinternals.com\tools via my (currently) 700kbs broadband and wifi:

    C:\test>net use l: Local name l: Remote name \\live.sysinternals.com\tools Resource type Disk The command completed successfully.

    The first time I download procexp.exe it takes 34 seconds for the 2.5 MB file:

    C:\test> prompt [$t] $p$g [ 4:41:09.11] C:\test>copy l:\procexp.exe . 1 file(s) copied. [ 4:41:43.33] C:\test>dir procexp.exe 11/07/2012 17:38 2,691,192 procexp.exe 1 File(s) 2,691,192 bytes 0 Dir(s) 101,746,388,992 bytes free

    After that it is almost instantaneous. This is done by hitting cursor up and enter as fast as I can:

    [ 4:56:14.29] C:\test>del procexp.exe && copy l:\procexp.exe . 1 file(s) copied. [ 4:56:14.34] C:\test>del procexp.exe && copy l:\procexp.exe . 1 file(s) copied. [ 4:56:14.38] C:\test>del procexp.exe && copy l:\procexp.exe . 1 file(s) copied. [ 4:56:14.42] C:\test>del procexp.exe && copy l:\procexp.exe . 1 file(s) copied. [ 4:56:14.47] C:\test>del procexp.exe && copy l:\procexp.exe . 1 file(s) copied.

    Between .04 and .05 of a second. That can only come from local caching.


    With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.

    The start of some sanity?

      I can't replicate that because of my residential cable ISP.
      No. Time Source Destination Protoc +ol Info 3 12.791678 192.168.1.112 157.56.203.150 ICMP + Echo (ping) request 4 14.956316 192.168.1.112 157.56.203.150 ICMP + Echo (ping) request 5 17.456431 192.168.1.112 157.56.203.150 TCP + 11975 > microsoft-ds [SYN] Seq=0 Win=16384 Len=0 MSS=1460 6 17.456837 192.168.1.112 157.56.203.150 ICMP + Echo (ping) request 7 17.465186 10.240.176.53 192.168.1.112 ICMP + Destination unreachable (Communication administratively filtered) 8 19.956352 192.168.1.112 157.56.203.150 ICMP + Echo (ping) request 9 20.441617 192.168.1.112 157.56.203.150 TCP + 11975 > microsoft-ds [SYN] Seq=0 Win=16384 Len=0 MSS=1460 13 22.456477 192.168.1.112 157.56.203.150 TCP + 11976 > netbios-ssn [SYN] Seq=0 Win=16384 Len=0 MSS=1460 16 22.465609 10.240.176.53 192.168.1.112 ICMP + Destination unreachable (Communication administratively filtered) 17 25.371335 192.168.1.112 157.56.203.150 TCP + 11976 > netbios-ssn [SYN] Seq=0 Win=16384 Len=0 MSS=1460 18 25.379983 10.240.176.53 192.168.1.112 ICMP + Destination unreachable (Communication administratively filtered) 19 26.376227 192.168.1.112 157.56.203.150 TCP + 11975 > microsoft-ds [SYN] Seq=0 Win=16384 Len=0 MSS=1460 20 31.405568 192.168.1.112 157.56.203.150 TCP + 11976 > netbios-ssn [SYN] Seq=0 Win=16384 Len=0 MSS=1460
      did a "net use \\live.sysinternals.com\tools" to generate the above. console says
      System error 67 has occurred. The network name cannot be found.
      A residential ISP can't be blamed for blocking SMB ports.

      update: Are you sure you are using SMB over the internet and not WebDAV? I turned on my Web Client service on XP, and retried to map the drive letter. I then ran
      use Win32API::File 'QueryDosDevice'; print $olTargetLen = QueryDosDevice( 'W:', $osTargetPath, [] ); print "\n\n"; print $osTargetPath;
      and got
      74 \Device\WebDavRedirector\;W:000000000000fce5\live.sysinternals.com\too +ls
      Thats not the SMB driver. If I rerun the script on an actual local network SMB share I get
      54 \Device\LanmanRedirector\;Z:000000000000fce5\removedservername\c
      and LanmanRedirector is the correct driver on NT for SMB.
        Are you sure you are using SMB over the internet and not WebDAV?

        Yes, I used WebDAV for the demo. I don't have an accessible other local machine right now. My other machine is taking a sabbatical.

        But I'm pretty sure that file shared across the local LAN were cached locally going right back to NT4. The first connections were always slow; and directory listings were never cached back then, so they were also slow; but individual file metadata was cached and reopening a remote file was always quicker the second time around.

        You are right about Vista/SMB2 though. Reading around, it seems to have improved things a lot through the use of bigger reads; coalesced commands and overlapped IO.


        With the rise and rise of 'Social' network sites: 'Computers are making people easier to use everyday'
        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.

        The start of some sanity?

Log In?
Username:
Password:

What's my password?
Create A New User
Node Status?
node history
Node Type: note [id://984555]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others examining the Monastery: (5)
As of 2019-03-22 01:50 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    How do you Carpe diem?





    Results (110 votes). Check out past polls.

    Notices?