It was funny and entertaining, but I don't really think it's non-geek friendly. I actually felt guilty for bringing the lady. She really enjoyed Iron Man, Nolan's Batman films, and Captain America (sidenote: I didn't even like Cap all that much), but this was just so full of other film's continuity and cosmic cubes and aliens...etc. The first thirty minutes or so were tough for even me to get through. Whedon's witty dialogue and the impressive action scenes will entertain the masses, but the film had plenty of holes. As an Avengers fan, I thought it was great. As a movie fan, I felt it was a lot of fun, but sort of a mess.
Edited out this last part. I don't know how to do spoiler tags on this forum. My bad.
I think it's perfectly enjoyable on a basic blockbuster level, the audience I saw it with audibly enjoyed it and I strongly doubt that they were all geeks. It does certainly benefit if you're a fan of the previous movies, Joss Whedon and of course the comics.
To me some of the standouts were the Helicarrier taking off and going into stealth mode (such a comic booky thing to make it into a movie), all the Hulk/Banner scenes (by far the best use of the Hulk on screen to date) especially Banner's "secret" line, all the face offs between the team and that tracking shot during the final battle going between the Avengers. I loved all the cool comic book stuff that made it into the movie, Hulk not being able to lift the hammer, Iron Man bouncing his repulsors off of Cap's shield to hit another foe, the x-ray shot of Hawkeye's quiver - I love that we're at the point where that stuff can be in a big mainstream movie, rather than the impression that the studio is embarrassed by the source material.
The only real weak point for me was the aliens, they were bland and non-threatening, but I understand why the focus of this first movie had to be the team themselves so they basically just needed to be generic punching bags - I expect we'll get a more significant foe next time around (I don't think it'll necessarily be the teaser though, though I'd be happy if it was). I'd also point out that plot wise the final act was almost exactly the same as Bayformers 3, but of course much better done.
The big question I have is how the solo sequels will compete. So much of what made this film work was seeing the major characters play off each other, if they just go back to doing their own thing I think the audience will really feel that abscence.
Now roll on Dark Knight Rises!
And yes Loki was the impetus for the Avengers forming in the original comics too.