Smarty Forum Index Smarty
WARNING: All discussion is moving to https://reddit.com/r/smarty, please go there! This forum will be closing soon.

Warning: filemtime($compile_path): Stat failed
Goto page 1, 2  Next
 
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> Bugs
View previous topic :: View next topic  
Author Message
RealX
Smarty Rookie


Joined: 02 Jan 2006
Posts: 9

PostPosted: Mon Jan 02, 2006 2:16 pm    Post subject: Warning: filemtime($compile_path): Stat failed Reply with quote

Hi, I'm using Smarty 2.6.11 (default config: disabled cache and disabled force compile) and I get the following warning sometimes:

Warning: filemtime(): Stat failed for /hosting/*****/templates_c/%%2E^2EF^2EFC11E6%%pagetemplate%3Amain.tpl.php (errno=2 - No such file or directory) in /hosting/*****/Smarty/Smarty.class.php on line 1392

Code:

                if ($_params['resource_timestamp'] <= filemtime($compile_path)) {
                    // template not expired, no recompile
                    return true;
                } else {
                    // compile template
                    return false;
                }



In the is_compiled function the compile_path is already checked by file_exists. This means that the warning is not possible. Confused

I added the file_exists($compile_path) in the if statement like this:

Code:

if (file_exists($compile_path) && $_params['resource_timestamp'] <= filemtime($compile_path))


This solution fixed the warning. Smile But why do I need to add file_exists($compile_path)?
Is it a PHP bug?
Back to top
View user's profile Send private message
messju
Administrator


Joined: 16 Apr 2003
Posts: 3336
Location: Oldenburg, Germany

PostPosted: Mon Jan 02, 2006 3:24 pm    Post subject: Reply with quote

I think it's a race condition between line 1382 (file_exists($compile_path)) and line 1392 (filemtime($compile_path)).
if another process removes the compiled template while the process is running in between these lines this error you mention occurs.

do you remove the compiled templates often?
Back to top
View user's profile Send private message Send e-mail Visit poster's website
RealX
Smarty Rookie


Joined: 02 Jan 2006
Posts: 9

PostPosted: Mon Jan 02, 2006 7:14 pm    Post subject: Reply with quote

No, I don't. Why you're asking?

I just noticed that I used a MySQL TIMESTAMP (YYYY-MM-DD HH:MM:SS) from database resources instead of a UNIX_TIMESTAMP. The templates were recompiled on every invocation. Embarassed
Now I use a UNIX_TIMESTAMP just like the returned value of filemtime(). The warning seems to be gone. I think this is the only solution...
Back to top
View user's profile Send private message
Milkman
Smarty Regular


Joined: 21 Nov 2005
Posts: 57
Location: Loughborough, UK

PostPosted: Wed Jan 04, 2006 1:17 am    Post subject: Reply with quote

messju, is there any file locking in the compiling or caching code to prevent more than one process from trying to generate, delete or update a file at anyone time? Something to let other requests for the same file know that someone else is doing something with the file and to wait a moment and try again. I realise this would add a few more file_exists calls but could be a security option (if it does not already exist)?
Back to top
View user's profile Send private message Visit poster's website
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Wed Jan 04, 2006 6:39 am    Post subject: Reply with quote

Hi.

AFAIK, you can't file_exists the race away because you are just pushing the issue up a level; now you have to contend with the issue of making the "busy signal" have a reliable locking scheme across request boundaries. ...which is basically the advisory lock provided by [url=php.net/manual/en/function.flock.php]PHP's flock()[/url] -- itself of no practical help, IMO.

http://cvs.php.net/viewcvs.cgi/smarty/libs/internals/core.write_file.php?view=markup&rev=1.2

That's the relevant code -- you can see Smarty takes steps to alleviate the issue, but I somewhat doubt it eliminates it. It may be worthwhile to consider avoiding the unlink() on non-windows systems and perhaps adding a random delay before the unlink() but I am not aware of a way to make the unlink/rename/chmod atomically (at least in the context of PHP userland and an arbitrary filesystem on every platform).
Back to top
View user's profile Send private message
Milkman
Smarty Regular


Joined: 21 Nov 2005
Posts: 57
Location: Loughborough, UK

PostPosted: Wed Jan 04, 2006 7:00 pm    Post subject: Reply with quote

If it is a windows only problem would it be possible to add a test for the os into the code? Though I'm guessing you don't want to go down those lines? It might make the code more secure for *nix systems?
[php:1:57c66ac9ba]
if (PHP_OS == 'Windows' && file_exists($params['filename'])) {
@unlink($params['filename']);
}[/php:1:57c66ac9ba]

Or perhaps another code change to consider might be:
[php:1:57c66ac9ba]
if (!@rename($_tmp_file, $params['filename'])) {
@unlink($params['filename']);
@rename($_tmp_file, $params['filename'])
}
@chmod($params['filename'], $smarty->_file_perms);[/php:1:57c66ac9ba]

Anyway, I'm sure you've thought of all these anyway, and I shouldn't hijack someone's bug report. Wink
Back to top
View user's profile Send private message Visit poster's website
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Wed Jan 04, 2006 7:31 pm    Post subject: Reply with quote

Hi again.

Quote:
Anyway, I'm sure you've thought of all these anyway, and I shouldn't hijack someone's bug report


I think there is merit in discussing it Smile

It is not a windows-only problem. It can happen if you are using a remote FS (say NFS) and with various other filesystems/kernels. It can also happen if you are using a threaded webserver or if you happen to have a non-web based user managing files (eg: through a cron job or dedicated recurring service). So there is no simple, general fix.

That said:
[php:1:3d80c9867d]
if (PHP_OS == 'Windows' && file_exists($params['filename'])) {
@unlink($params['filename']);
}
[/php:1:3d80c9867d]
is what I was proposing except that it still does not fix the issue in any way -- it just avoids an unlink on non windows platforms. One thing to consider, though, is adding code to help stop corrupted files from being created:

[php:1:3d80c9867d]
// don't allow PHP to quit even if the user aborts
ignore_user_abort(true);

// a small sleep to avoid contention would be useful here if we
// used flock() -- but we don't

// here the file_exits() may be superfluous since @unlink() is used
if (PHP_OS == 'Windows' && file_exists($params['filename'])) {
@unlink($params['filename']);
}
// however, another process may have written the filename at this point,
// thus the following rename can still fail on windows

@rename($_tmp_file, $params['filename']);
@chmod($params['filename'], $smarty->_file_perms);

// allow PHP to quit if the user aborts
ignore_user_abort(false);
[/php:1:3d80c9867d]
Back to top
View user's profile Send private message
Milkman
Smarty Regular


Joined: 21 Nov 2005
Posts: 57
Location: Loughborough, UK

PostPosted: Thu Jan 05, 2006 8:51 pm    Post subject: Reply with quote

I think my second bit of code might be better as the problem is not only windows. But still does not solve the problem. I also agree that adding in a lock file won't help brilliantly as if the call to file_exists happens at the same time as a touch call the file won't exist. And I don't see the rename command using the file for long enough to make it worth file locking.

Out of curiosity, what is likely to happen if two threads/processes are renaming to the same filename at the same time? Does php prevent both files from merging? Would it be worth hashing the tmp file and checking the hash afterwards?
Back to top
View user's profile Send private message Visit poster's website
c960657
Smarty Regular


Joined: 07 May 2003
Posts: 75
Location: Copenhagen, Denmark

PostPosted: Fri Oct 20, 2006 9:28 am    Post subject: Reply with quote

Milkman wrote:
I think my second bit of code might be better as the problem is not only windows. But still does not solve the problem.

As long as it solves the problem on some platforms (and doesn't hurt the rest), it is a step in the right direction.

boots, I don't understand why you don't consider using flock() an option?
Back to top
View user's profile Send private message Visit poster's website
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Fri Oct 20, 2006 3:51 pm    Post subject: Reply with quote

c960657 wrote:
boots, I don't understand why you don't consider using flock() an option?


I'm not entirely opposed to it but it isn't a very robust solution. You really need to be able to control your environment (and pick-and-choose appropriate components) for it to actually work -- and even then, only so long as all of your processes on that same box are coordinated to use advisory locks. So while we can implement flock() and cause additional overhead for all users -- only some users will get benefits from it. IMO, you need real transactional support (probably with separate writer and reader threads/processes) to make a real go of this -- and that's just not feasible with the majority of FS's and the way PHP is used in webserver processes.

Of course, this is one of those things where I would gladly rather be proven wrong on the issue Smile
Back to top
View user's profile Send private message
messju
Administrator


Joined: 16 Apr 2003
Posts: 3336
Location: Oldenburg, Germany

PostPosted: Fri Oct 20, 2006 5:09 pm    Post subject: Reply with quote

flock() would gain us nothing since it does not work in multithreaded environments. and coincidently the world where multithreaded servers seem to dominate is the same where rename() cannot transactionally overwrite.
Back to top
View user's profile Send private message Send e-mail Visit poster's website
c960657
Smarty Regular


Joined: 07 May 2003
Posts: 75
Location: Copenhagen, Denmark

PostPosted: Fri Oct 20, 2006 5:22 pm    Post subject: Reply with quote

As long as Smarty is the only one writing files in the templates_c directories, I wont be a problem that flock() is only advisory AFAICT.

But I agree that the performance overhead using locking may be overkill.

On systems that support overwriting using rename(), we can simply skip the unlink() call and be happy.

On systems that does not overwriting using rename(), e.g. Windows, we can compare the contents of the file with the contents we are about the write and only call unlink() if the file already exists and its contents differ (apart from the timestamp that is added to the compiled templates). Only in this case, i.e. when a template has changed (or some pre/postfilter, compiler function etc. makes the compiled template output depend on other things than the template input, e.g. the current time), unlink() may cause errors.

What I am suggesting is something like this (not tested):
Code:
if (!@rename($_tmp_file, $params['filename'])) {
    $fd = @fopen($params['filename'], 'rb');
    $old_contents = $fd && ($size = filesize($params['filename'])) ? fread($fd, $size) : '';
    if ($fd) {
        @fclose($fd);
    }
    if ($old_contents != $params['contents']) {
        @unlink($params['filename']);
        @rename($_tmp_file, $params['filename'])
    }
}

This code assumes that core.write_compiled_include.php no longer writes the compile time to the header of the compiled template. Another solution is to skip the first X characters when comparing the strings.
Back to top
View user's profile Send private message Visit poster's website
boots
Administrator


Joined: 16 Apr 2003
Posts: 5611
Location: Toronto, Canada

PostPosted: Fri Oct 20, 2006 6:10 pm    Post subject: Reply with quote

c960657 wrote:
As long as Smarty is the only one writing files in the templates_c directories, I wont be a problem that flock() is only advisory AFAICT.
No, that misses the point. Several instances of Smarty can be reading/writing and that is the very problem -- because flock() is not thread-safe. Even if it is "only" Smarty doing the work, it still looks like many un-coordinated readers/writers to the fs. "Advisory" is, at best, a suggestion. Besides, I would think that I am not alone in using chron jobs and custom services that do routine maintenance and data handling against Smarty sources.

IMHO, this problem is a lot harder than it sounds and there are no simple solutions. It seems to me that the best that can be achieved is mitigation and the challenge is balancing the amount of mitigation that is done versus the cost of taking such steps.

Being able to plug the filesystem handlers would help. Then someone could implement a handler that uses an transaction resource (such as a db). Perhaps this can already be done in a stream wrapper -- but I doubt it.
Back to top
View user's profile Send private message
c960657
Smarty Regular


Joined: 07 May 2003
Posts: 75
Location: Copenhagen, Denmark

PostPosted: Fri Oct 20, 2006 6:32 pm    Post subject: Reply with quote

boots wrote:
No, that misses the point. Several instances of Smarty can be reading/writing and that is the very problem -- because flock() is not thread-safe.

Ah, crap! I thought it implemented a true file lock as long as all processes accessing the file in question were using flock() before writing/deleting the file. But okay, let's just forget about flock() then.

boots wrote:
IMHO, this problem is a lot harder than it sounds and there are no simple solutions. It seems to me that the best that can be achieved is mitigation and the challenge is balancing the amount of mitigation that is done versus the cost of taking such steps.

AFAICT the problem can be fixed completely on the systems that support it - which is probably a large percentage - if we only call unlink() if rename() returns false. This is a simple change that has little if any performance impact.
Back to top
View user's profile Send private message Visit poster's website
c960657
Smarty Regular


Joined: 07 May 2003
Posts: 75
Location: Copenhagen, Denmark

PostPosted: Mon Nov 06, 2006 2:28 pm    Post subject: Reply with quote

Okay, here is a concrete suggestion that removes the problem on some systems and leaves status quo on the rest:
Code:
@@ -36,16 +36,16 @@ function smarty_core_write_file($params,
     }

     fwrite($fd, $params['contents']);
     fclose($fd);

-    // Delete the file if it allready exists (this is needed on Win,
-    // because it cannot overwrite files with rename()
-    if (file_exists($params['filename'])) {
+    if (!@rename($_tmp_file, $params['filename'])) {
+        // On platforms and filesystems that cannot overwrite with rename()
+        // (e.g. Windows), delete the file and try again
         @unlink($params['filename']);
+        @rename($_tmp_file, $params['filename']);
     }
-    @rename($_tmp_file, $params['filename']);
     @chmod($params['filename'], $smarty->_file_perms);

     return true;
 }



Do you see any problems with that? I hasn't been tested on Windows (yet).
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
This forum is locked: you cannot post, reply to, or edit topics.   This topic is locked: you cannot edit posts or make replies.    Smarty Forum Index -> Bugs All times are GMT
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group
Protected by Anti-Spam ACP