Skip Menu |
 

This queue is for tickets about the CHI CPAN distribution.

Report information
The Basics
Id: 76746
Status: resolved
Priority: 0/
Queue: CHI

People
Owner: Nobody in particular
Requestors: vitaliy.tokarev [...] gmail.com
Cc:
AdminCc:

Bug Information
Severity: Wishlist
Broken in: 0.52
Fixed in: (no value)



Subject: CHI::Driver::File - speed up get_keys() function
I use File driver in situations when I have a small amount of keys, but I often call get_keys(). Even when we use L1_CACHE, get_keys() calls find(). More over, the subdirs is never deletes via this driver. So, we call often get_keys() which calls find() on every subdir in the directory where keys stored. I wrote a micro patch, which can be useful for people with same situation. I don't know about overhead on scalar function, but I consider it is better then we are calling filesystem functions every time. I put here full code to easy understanding and discuss: my $is_empty = 0; sub get_keys { my ($self) = @_; return () if ($is_empty); my @filepaths; my $re = quotemeta($self->file_extension); my $wanted = sub { push( @filepaths, $_ ) if -f && /${re}$/ }; my @keys = $self->_collect_keys_via_file_find( \@filepaths, $wanted ); $is_empty = (scalar @keys) ? 0 : 1; return @keys; } Much better will be store results somewhere and just look for modifications, but then in situations where a lot of keys CHI will eat more memory. What are you say about this patch?
From: vitaliy.tokarev [...] gmail.com
Download (untitled) / with headers
text/plain 147b
Also, if someone will use this patch, dont forget to reset $is_empty via store() function: sub store { ... write_file(...); $is_empty = 0; ... }
Subject: Re: [rt.cpan.org #76746] CHI::Driver::File - speed up get_keys() function
Date: Sat, 21 Apr 2012 21:55:38 -0700
To: bug-CHI [...] rt.cpan.org
From: Jonathan Swartz <swartz [...] pobox.com>
Download (untitled) / with headers
text/plain 1.9k
This won't work in a situation with multiple processes accessing the same file cache - imagine that $is_empty is set in process 1, then process 2 stores a new key; process 1 will still think there are no keys. Best thing might be to have a periodic process that deletes empty directories or old files in your cache file hierarchy. Jon On Apr 21, 2012, at 10:10 AM, Vitaliy Tokarev via RT wrote: Show quoted text
> Sat Apr 21 13:10:49 2012: Request 76746 was acted upon. > Transaction: Ticket created by gh0stwizard > Queue: CHI > Subject: CHI::Driver::File - speed up get_keys() function > Broken in: 0.52 > Severity: Wishlist > Owner: Nobody > Requestors: vitaliy.tokarev@gmail.com > Status: new > Ticket <URL: https://rt.cpan.org/Ticket/Display.html?id=76746 > > > > I use File driver in situations when I have a small amount of keys, but > I often call get_keys(). Even when we use L1_CACHE, get_keys() calls > find(). More over, the subdirs is never deletes via this driver. > > So, we call often get_keys() which calls find() on every subdir in the > directory where keys stored. I wrote a micro patch, which can be useful > for people with same situation. I don't know about overhead on scalar > function, but I consider it is better then we are calling filesystem > functions every time. > > I put here full code to easy understanding and discuss: > > my $is_empty = 0; > > sub get_keys { > my ($self) = @_; > > return () if ($is_empty); > > my @filepaths; > my $re = quotemeta($self->file_extension); > my $wanted = sub { push( @filepaths, $_ ) if -f && /${re}$/ }; > my @keys = $self->_collect_keys_via_file_find( \@filepaths, > $wanted ); > $is_empty = (scalar @keys) ? 0 : 1; > return @keys; > } > > > Much better will be store results somewhere and just look for > modifications, but then in situations where a lot of keys CHI will eat > more memory. What are you say about this patch?
From: vitaliy.tokarev [...] gmail.com
Download (untitled) / with headers
text/plain 984b
Yes, you are right. Ticket may be closed due specific of a situation. We are trying to resolve our problems and this was my mistake. Some months ago I wrote the program that looking for modifications of files, event-based (EV module). And I may say that is not best practice, because if we have a lot of files EV could not solve this task ideally. So, this is an old programming problem what to choose: of cpu time or a memory usage. Memory is cheap, cpu is not still. I will try find best solution for this question and post here if luck is still with me. Thanks! On Sun Apr 22 00:55:54 2012, swartz@pobox.com wrote: Show quoted text
> This won't work in a situation with multiple processes accessing the > same file cache - imagine that $is_empty is set in process 1, then > process 2 stores a new key; process 1 will still think there are no > keys. > > Best thing might be to have a periodic process that deletes empty > directories or old files in your cache file hierarchy. > > Jon


This service is sponsored and maintained by Best Practical Solutions and runs on Perl.org infrastructure.

Please report any issues with rt.cpan.org to rt-cpan-admin@bestpractical.com.