I'll be honest, when I first saw AGAL some years ago, I just knew it was too obtuse and complicated for me. I was sure they'd make it easier in the next release, but 3 years later I'm still waiting.
Time to take the plunge. There's loads of great articles about AGAL and how to set it all up. It requires a ton of boiler plate AS3 code before you even get to the shader programming, but that part is at least just about identical every time so after a couple hours of wading through it line by line, you don't need to think about it any more.
I've made shaders in PixelBender, so the concept isn't entirely new to me, but in pixel bender you write a routine which is executed on each pixel. In AGAL you write a routine that is executed first on every vertex, and then on every pixel (or fragment) between them.
One of the really annoying things is the way you have to declare any variables you're going to use outside of the actual program. So you end up hopping around between shader code and AS3. The next hurdle is the actual names of the variables, as you don't get to name them yourself. vc0 for example is a vertex constant in register 0, but rather than assigning something to vc0 (vc0 = would have been nice!) you would write something like..
context3D.setProgramConstantsFromVector(Context3DProgramType.VERTEX, 4, sway, -1);
Believe it or not, that has assigned a vector ("sway") to vc0.
As they say in the Matrix, "after a while you don't even see the code". I wish!
With all the register indexes and two letter names it is very easy to make mistakes.
I'm not going to attempt to explain the mechanics, as it's been done to death and way better than I could.
I recommend starting with http://blog.norbz.net/tutorials/ and if you can get hold of this book, all the better.
And this is your Bible:
That will get you up and running, but then what? There's simply tons of articles about shader programming out there, but precious few about AGAL specifically.
For a bit of inspiration I recommend a trip to shadertoy.com, but be warned, your browser will probably crash a few times, and your laptop will melt. These are GLSL shaders for WebGL, so while they may give some ideas it's far from obvious how to implement them in AGAL, and once you get past the basics of Stage3D there's not much help around.
An excellent free resource is the GPU Gems series, hosted at Nvidia.
This evening's experiment has been implementing this little ditty, animated grass.
These books are great, as they explain the concepts, rather than the code.
One of the most beautiful sites you'll ever see in Denmark is a stiff sea breeze causing waves in a ripe field of barley. That was what I decided I'd try to emulate.
As I was working with an example that had different Blend modes, I left them in (After clicking on the window use "B" to switch). I also wanted to explore and understand the mip mapping modes, so you can go through all those by pressing SPACE. Oh, and to prove it's not baked, you can use your mouse to effect the grass too.
Frame rate varies wildly. There are exactly 65536 polygons in this example, all being animated independently in groups of 16 (i.e. a single blade of barley).
There are definitely more efficient ways, but it runs at 60fps on my laptop. I can see online its not quite so good.
What kills performance here is not the shader, but the fact I'm reading in Perlin noise from a BitmapData animated in Flash for every cycle. I'm sure there's a better way, but it's late enough as it is...