[Da:Sourcerer]

This is related to ticket #313: I've been working for about a month on a http client for Yii. The ticket has been attached to the 2.0 milestone, but I think it can be done in 1.1.x already.

I fetched some inspiration from the Zend_Http_Client class and Habari's remote request class. Work on this is far from complete and this component has been very little tested, so please don't use this as-is!

https://github.com/D...CHttpClient.php

What's still missing:

• unit test coverage
  
• cookie treatment
  
• http caching
  
• ci treatment of headers (as per RFC)
  
• complete proxy support
  
• supporting code for POST requests

Edit: Updated link

[Haensel]

Very nice. I can't count the number of times I missed a component like that. Can't give you a thumbs up via the forum's mobile theme though ;-)

Edit: Interesting to see that you aren't using php curl. So this will effectively work without any additional dependencies, very nice. I guess I'll will swap my "ActiveResource" request and response classes in favor of your work as soon as it is released. They totally suck compared to this

[Da:Sourcerer]

Haensel, on 04 September 2012 - 10:59 AM, said:

Can't give you a thumbs up via the forum's mobile theme though ;-)

There's a mobile theme?

Haensel, on 04 September 2012 - 10:59 AM, said:

Interesting to see that you aren't using php curl. So this will effectively work without any additional dependencies, very nice.

That's intended. The component should use as little resources as possible (that's part of the reason why you cannot get the raw response body). However: Once I've finished, it should be no problem to set a CHttpClientCurlConnector in place to let all the connectivitiy stuff pass through the cURL extension

[Da:Sourcerer]

A console command as a usage example:

class HttpCommand extends CConsoleCommand
{
	public function run($args)
	{
		$target='http://kernel.org';
		if(isset($args[0]))
			$target=$args[0];
		
		$response=Yii::createComponent('CHttpClient')->fetch($target);
		print_r($response);
	}
}

That's pretty much what I'm using for testing atm.

[Da:Sourcerer]

Short status update: Work on the component continues and is closing in on a usable state, the request and response classes are now covered by unit tests (>70% for both). Http compression support has drastically improved and there now is very basic support for cookies.

What's currently happening: I'm rewriting some of the stream socket stuff to revolve around stream_context_create() and stream_socket_client(). It looks like I won't be able to access the finer points of ssl-protected connections otherwise. This will also mean that the default connector class will resemble Zend Framework's stream socket adapter much more closely than I intended. On the pro side, this will eliminate the need for the proxy connector, reducing code complexity greatly.

A weak point remains file up- and downloads: They have to be kept entirely in memory for the time being. The request and response classes are also not intelligent enough yet to handle them for you: You've got to do all the parsing and encoding yourself.

A great future addition were probably the implementation and support for HTTP pipelining, which will be great if remote resources should be worked on by batch. I do not dare yet to touch that: CHttpClient will probably become the most complet http client I've written so far, so there are other hot issues now. However, it will definitely be something I'll look at if (when) this component will be ported to Yii 2.

All in all, a lot of progress is being made and if I keep up the pace (i.e. I don't run out of coffee) things are looking good for a working CHttpClient in Yii 1.1. In addition^*, I plan to release this component as an extension for projects which are stuck with older framework versions.

*) That's also plan b

[Da:Sourcerer]

I'd like to fetch some more input before I open up a pull request. I plan to finish up test coverage as well as the cookie- and header-management code this week. So if there are any weak spots in the API, it would be nice if you would tell me now.

The current state is this: About every imaginable HTTP compression scheme is supported now: Raw deflate, gzip, bzip2 ... all there. The currently only connector is capable of reading in chunk-encoded content on its own now, which is great for PHP < 5.3 which doesn't seem to ship with the dechunk stream filter. All of the network stuff is now handled through stream_socket_client() and stream_context_create(). This means there is better control over the behaviour of SSL-enabled connections now. Full proxy support is also achieved through this. The code for dns resolving is gone for now. I plan to patch said code into CUrlValidator so the host-part of URLs can be validated through this (similar to what CEmailValidator can do atm).

The code has matured to a pretty usable state. If you understand it well enough, you can sue it to fetch pretty much every website and even query webservices. E.g. everything is there to pull some information out of GitHub's JSON-based API. Redirects are a bit wank, but they mostly work.

Items left to do are:

• cookie parsing and storage
  
• http caching
  
• helpers for query strings
  
• finishing up the phpdoc

Handling downloadable files is still an issue as all responses will be held in memory. Juts looking for the presence of the Content-Disposition http header is no good metric. Same goes for uploading files: They need to be loaded into a request object that holds the entire request in memory. Handling this will require some additional thought and will probably not come until a re-implementation in Yii 2. Zend Framework 2 handles this by making the response and request objects "streamable". I'm hesitant to go that way right now since this will require some infrastructural changes and might prevent this from getting released with v1.1.13. Another thing that I'll postpone for now is support for HTTP authentication schemes beyond "Basic". It's not that hard to implement, but I'd like to present a working component first and patch this in later. Future work will include a connector based on cURL (as mentioned earlier) and one using pecl::http.

[Da:Sourcerer]

So, after an involuntary break I've pcked up work on CHttpClient again. Having missed the 1.1.13 release gave me the opportunity to meditate on the written code so far. Having done so (and having extended my research on Guzzle, Requests for PHP and the libcurl internals), I am now in the process of commencing a large-scale rewrite. The goal will be to allow a much nicer API allowing method chaining quite similar to CDbCommandBuilder etc. I also had to learn that I had been a bit naïve in regards to the complexity of cookie and header parsing, let alone efficient management of those. In this regard I'd like to fetch some input on how to solve the problem of header management as I certainly do not wish to copy the approach Zend Framework 2 has taken.

Another hot topic is the usage of temporary PHP streams. They were introduced in PHP 5.1 and are held in memory until they pass a certain threshold or the memory_limit is being hit, in which case they are dumped to the HDD. Temp streams are used heavily by Guzzle and they solve quite a lot of problems I earlier had. E.g. they make it far less dangerous to download files of unknown size. Unfortunately, they will also hamper interoperability with pecl::http and libcurl. We will see how well this goes.

Since I've found myself dealing with URL manipulations a lot more than anticipated, I see the need for a CUrl object arising. I heavily opposed that idea here. Although that happened for different reasons, you know what they say about peopl who never change their oppinion

So, all in all this turns out to be a lot more work than I initialy thought. Mostly thanks to server quirks who are a lot harder to come by than client quirks

In the meantime, feel free to use Guzzle. It is ridiculously easy to set up with Yii and I expect the result of my work to be similar regarding API calls.

[fr0d0z]

Thanks for working so hard on this. Not having looked at Guzzle much yet, any chance you can build on top of it instead of having to write everything yourself? I noticed they're integrating Guzzle into Drupal 8, so I imagine it will be around for a while.

Thoughts? If we can have someone else do the heavy lifting and not reinvent the wheel, seems like it could help ...

[Da:Sourcerer]

Short answer: No. Guzzle is relying heavily on PHP 5.3 features such as namespaces. As I see it, this is an unsolvable conflict with Yii's bc-breaking policies.

[jacmoe]

Yii 2.0 is requiring PHP 5.3+

[Da:Sourcerer]

True. But we're still talking about 1.1 here, aren't we?

[fr0d0z]

:-) Yeah, I noticed the PHP 5.3 requirement for Guzzle after I posted, sorry about that.

That leads to a different question. Is it worth the effort of building this for v1.1? Having looked over Guzzle more closely, there is a lot there that looks pretty nice. I wonder if it would be worth making Guzzle the default for Yii 2, as it will be built-in for Drupal 8 and may make it far easier for folks to make the jump from Drupal down the road (and may ease tying in with Drupal systems as well).

It looks like Guzzle could be gearing up to be a library used in a lot of places and I wonder if we'd be best served by getting on board. Thoughts?

[jacmoe]

Suggesting Guzzle - although it looks really nice - is defeating the purpose of this effort (if I'm understanding it right):

Quote

All the power of cURL with a simple interface.

Yes, it requires cURL.

In fact, it requires:

Quote

PHP 5.3.2+ compiled with the cURL extension
A recent version of cURL 7.16.2+ compiled with OpenSSL and zlib

[fr0d0z]

I think I'm tying myself up in knots, so maybe I should stop. But one last try :-)

I really appreciate Da:Sourcerer's efforts (and don't want to discourage him for making them, as I'd use it if it was available). Just was wondering if the benefit is worth the effort long-term? Guess that may be more up to him to decide ;-)

[jacmoe]

Who needs purpose?

I mean, as long as there is a clear motivation behind it. Which seems to be a minimal, but powerful, Http client with no dependencies (at least not on cURL).

If cURL was an option, it would be really easy. There are several cURL extensions/helpers/whatnots for Yii 'out there'.

[fr0d0z]

jacmoe, on 07 March 2013 - 12:06 PM, said:

Who needs purpose?

I stand corrected ;-)

[Da:Sourcerer]

@jacmoe: You're wrong about cURL. cURL is one of in fact three methods to establish connectivity within Guzzle. The other two involve the native php http wrapper and the pecl:http package.

And yes, I'm a bit selfish because I like the intellectual challenge of this. This isn't my first HTTP client in PHP but by far the most complete as I am not writing it for a specific task (such as fetching a remote image or polling an API endpoint) but designing and implementing it with a general purpose in mind. Otherwise it were just too easy to copy the http implementations of Zend Framework 1, Symfony or Horde over to Yii

[jacmoe]

Guzzle says that it requires cURL: http://guzzlephp.org...stallation.html
And a pecl package is an even worse requirement, IMO.
The third option would probably require PHP 5.4 ? I'll read the docs.

But I understand totally why you're doing this.

[Da:Sourcerer]

Hm, fascinating. I really don't know why it's being listed as a requirement. My understanding of the code is that it can operate entirely without cURL being installed.

[fr0d0z]

I read through the Drupal 8 / Guzzle adoption thread last night:
http://drupal.org/node/1447736

Guzzle does require cURL (per mtdowling, the author):
http://drupal.org/no...comment-6497932
http://drupal.org/no...comment-6501878

And the Drupal folks have been going back and forth about a cURL requirement on that thread, but they settled on requiring it. With them adopting it, I think we'll see cURL get even wider adoption than it already has (according to them). They mentioned that Wordpress has built their own HTTP library though to prevent a cURL requirement.

Might be interesting to run a poll on the forum to see how many folks currently using cURL have access to it on any shared servers they have. I've got some clients on Dreamhost and per that thread, Dreamhost already has it installed. I can check on some other hosts if we want to pursue this further.