Re: Working with large datasets

["user080701" <user080701@xxxxxxxxxxx>]

--------------------------------------------------
From: "Jack van Zanen" <jack@xxxxxxxxxxxx>
Sent: Tuesday, October 11, 2011 12:09 PM
To: "Jason Pruim" <lists@xxxxxxxxxxxxxxxxxxxx>
Cc: "Thompson, Jimi" <jimit@xxxxxxxxxxxxxxxx>; "Bastien" 
<phpster@xxxxxxxxx>; <php-db@xxxxxxxxxxxxx>
Subject: Re:  Working with large datasets

> Hi
>
>
> You need to index the right fields. even on a laptop a select from 8 
> million
> rows with two rows returned should take a few seconds max only.
> The first time you run the query the data has to come from disk, second 
> time
> you run same query you'd expect that data to sit in cache and be very 
> quick.
>
>
> Jack van Zanen
>
> -------------------------
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient. If you are not the intended recipient,
> please be aware that any disclosure, copying, distribution or use of this
> e-mail or any attachment is prohibited. If you have received this e-mail 
> in
> error, please contact the sender and delete all copies.
> Thank you for your cooperation
>
>
> On Tue, Oct 11, 2011 at 10:39 AM, Jason Pruim 
> <lists@xxxxxxxxxxxxxxxxxxxx>wrote:
>
> > Jason Pruim
> > lists@xxxxxxxxxxxxxxxxxxxx
> >
> >
> >
> > On Oct 10, 2011, at 5:27 PM, Thompson, Jimi wrote:
> >
> > > I really think that you should try running it from the command line and
> > see what the issues are.  Get both Apache and php out of the way.  I've 
> > seen
> > some PHP scripts use up all the file handles (OS limit) even on a 64 bit
> > server when they start doing complex things with data sets.
> > >
> > > If it works ok without PHP/Apache then you can start lookig at PHP and
> > APache.
> > >
> > > ISOLATE the issue not complicate it....
> > >
> > > My 2 cents,
> >
> > Hi Jimi,
> >
> > I've done it from the command line a few times, first time I run a 
> > simple:
> > SELECT * FROM Main WHERE state="test"; it takes:  2 rows in set (1 min 
> > 44.20
> > sec)
> >  after that initial run it's pretty responsive, usually around 0.01
> > seconds.
> >
> > If I do a select based on the new-york state, which is all of my almost 9
> > million records after the initial run, it returns it all VERY quickly:
> >  25000 rows in set (0.02 sec)
> >
> > So commandline is running fine after the initial...  I'm leaning towards
> > either a problem with web server, PHP setup, or my PHP code... I've used 
> > the
> > code many times before but never on such a large dataset...
> >
> > When I pull the pagination out completely.... It's pretty much the same
> > result... fetching the info for new-york works just fine but not "test"
> >
> > One thing I am noticing right now though is the fact that when I switch
> > over to using the test state.... Right now it's not displaying 
> > anything...
> > Not even able to view the source...
> >
> > Okay... Enough rambling right now... Need to do some more checking before 
> > I
> > can come to a conclusion :)
> >
> >
> >
> > --
> > PHP Database Mailing List (http://www.php.net/)
> > To unsubscribe, visit: http://www.php.net/unsub.php
>
>
>
> __________ Information from ESET NOD32 Antivirus, version of virus 
> signature database 6532 (20111010) __________
>
> The message was checked by ESET NOD32 Antivirus.
>
> http://www.eset.com

__________ Information from ESET NOD32 Antivirus, version of virus signature database 6532 (20111010) __________

The message was checked by ESET NOD32 Antivirus.

http://www.eset.com




--
PHP Database Mailing List (http://www.php.net/)
To unsubscribe, visit: http://www.php.net/unsub.php