PEP Court

Paul has mentioned that we might need a PEP court some day.  A place where folks who had BBs rejected could make their appeals.

Since the entire program is free and the certifiers are volunteers, there's a strong desire to make it easy on the certifiers.  Edge cases, incomplete submissions or ones that require behind-the-scenes staff discussions gum up the works.    

There are probably several ways to create a PEP court.  Any ideas?

My idea would be:

Submitter can request a Level 1 Review for rejected submissions.  They can write up their reasoning and submit it.  It gets sent to at least 3 certifiers to confer, debate and decide the case (anonymously).  You can request this twice for free, third time is $20, fourth time is $40, fifth time is $60, etc.  To clarify, you can only do a Level 1 review once per BB submission.  So your two free ones could be used on a Food Prep BB and a Natural Building BB.  After that you're laying out coin for the process.

You can request a Level 2 Review.  It goes to Paul Wheaton for a decision.  First time is $200, second time is $400, third time is $600, etc.

My first thought was NO that's extra work, but then I had another thought, which is if there are more people certifying then there will be more variation in that process.

My idea would be that you can request a review that goes to a more experienced reviewer. (if your original submission was checked by one then you are out of luck) I would not charge but I would have a one review limit. for a second review the submission would need to be significantly changed.

I think the idea of a review/appeal is a good one.  But to avoid frivilous appeals, make even the 'free' first appeals more valuable:

Each of us can appeal a rejected BB two times.  Period.  Lifetime.  You get two appeals, use them wisely.  That'll eliminate the ones attempting to game the system.

Thanks Skandi and Gary!  Yes Skandi, as more and more people become reviewers, I can imagine more variation in what is approved or rejected.  Trying to keep track of which certifiers are more experienced might require a whole extra system behind the scenes.  Maybe that's easy, maybe it's hard, I'm not sure...  The idea of the 3 person Level 1 appeal is that some of those people would likely be more experienced.  They might have to be on staff in order to render judgement.

Gary, I clarified some wording in the Level 1 verbiage above.  My intent was that each person only gets two free Level 1 reviews for their lifetime.  Not two free reviews per BB if that's what you thought I was saying.  Is that what you meant?  If I cleared that up, did that resolve the "gaming the system" issue that you were seeing?

I'm also assuming that this system will become a mechanism that we point people to when they complain.  When they realize it will cost money if they use it, they'll start doing more complete/correct submissions.  Another part of a self-sustaining system that manages itself.

I think its a good idea to give people a second chance. mike you sure have a whole bunch of pep badges which means you have a whole lot of experience in many vast areas and vocations.
to some people affirmation means a lot.  to other affirmation is everything and existence without it seems pointless. and to others affirmation has little to no meaning as they plow though life. it can compel some to further heights in life to strive for more and varied experiences. its a good thing all around. and as a fund raiser the fees after two freebees sounds like a very fair way to help fund the site.
just my $0.02 from an old phart

Mike Haasl wrote: My intent was that each person only gets two free Level 1 reviews for their lifetime.  Not two free reviews per BB if that's what you thought I was saying.  Is that what you meant?  If I cleared that up, did that resolve the "gaming the system" issue that you were seeing?

I think we're on the same page.  Definitely not 2 free reviews per BB!  Two complimentary reviews per permie.  After that, open the wallet.  

Actually, I like two free reviews, *period*.   With no cash option.  If there's a permie willing to appeal a third, fourth, et al BB in exchange for cash, means that permie is doing a lot of tasks incorrectly, or cutting corners or something, and not using the review panel's advice to improve the permie's performance.

Ooooh, I just thought of this:  Two free appeals.  The third appeal, straight to Paul, $200.  There, that makes the high maintenance permie worth the fuss they cause.  That makes the two free appeals very valuable!  No frivilous appeals!  A time saver.

I like that approach too.  Some day there might need to be an incentive to get reviewers to do the Level 1 review.  Maybe the $20 could be split between the people that are willing to debate and cogitate on a edge case.  But your approach fixes that problem too.  

Anything that brings some coin to Paul to keep Permies going is a good thing.

Perhaps the right to appeal should only kick in once a person has a certain number of BB under their belt?  Like maybe at BB 10 or 20?

Sort if like the cider press, where you earn your way in.  A person who has already completed such a number of BBs has shown that they do understand the system and they are invested in their progress, their complaints may warrant more consideration.  That way moderators are not run ragged by demands from people who are not serious.

I say this as a person who is not terribly serious about PEP. I enjoy completing BB because I am challenging myself to learn new skills,  but I am pretty set in life and do not really need my the extrinsic verification of my skills. I do not want moderators burnt out by weighing the minutiae of submissions by people like me who are just doing BBs for fun.  

I'm just going to point out that the discussion so far seems to be mostly assuming that all the problems will come from those submitting their work for the BB's, which does seem like the most likely case.  However when there are more reviewers it is entirely possible they could genuinely become the problem due to less experience, personal bias or prejudice, etc.  I'm not sure what the system will be to determine who becomes a reviewer.  Hopefully it will naturally weed out these sorts of issues but it's possible stronger attention might need to be paid at some point to the quality of the reviewers.  (To be utterly clear, I have zero problems with all you who are reviewing things now!)

What I can easily picture happening is a strong inconsistency in what is approved and what isn't leaving someone constantly frustrated when they see their submissions over and over refused while submissions of others that seem to have all the same criteria are approved.  

If you  go with a 2 freebies thing it might be good to have some simple way in the computer system for Paul or a high level reviewer to be able to quietly and privately give someone they see has been unjustly wronged some "extra" freebie appeals.  I would still leave things publicly presented as only 2 allowed so people do consider them precious and don't use them frivolously.  In this case though I might not state that this would be a lifetime limit.  That could just be implied.  The problem with stating it is that if you then had situations where it was fair and just to give someone more you'd likely end up with others declaring foul that they didn't get them, and so on.  

Great points David!  I think the current plan is that to become a certifier for a particular BB, you need to have been certified for that particular BB and have gotten the badge that the BB is associated with.  

maybe PIE is somehow involved?

paul wheaton wrote:maybe PIE is somehow involved?

The right to appeal could be a PIE feature. It's really cool that SKIP is free to anyone. A slice of PIE could be bought to help pay for the additional effort of an appeals process. And, by habitually being an excellent poster a person can even still get PIE without having to buy it.

I think the most important feature we would need is the "DUN-DUN!" sound effect from that show, what is it, Law and Order? Hopefully it's not copyrighted.

If a revenue stream is the goal, them PIE seems an easy implementation, and might enhance the user's Permies experience in other ways... there's also a way for it to be "earned" by receiving PIE as a gift. Maybe the one reviewing the appeal gets the slice of PIE? Maybe your Nth appeal costs "N pieces of PIE"?

I like the BB10, 20, 30, etc... idea where one appeal, or the ability to appeal, is earned at each interval, so there's some sweat equity and a track record to establish. This would weed out the newbies appealing their first, second, third BB's.

As far as concerns over "who's watching the watchmen?", the idea of experienced certifiers (already having done that BB and badge) seems like a decent start. Bumping appeals up to more senior reviewers, staff, or Paul seems logical.
Maybe the certifier who rejected the BB also gets reviewed as part of the appeal? Either they get a pat on the back, or learn how to better certify a BB/this BB? at the same time.

What if you had just volunteers to do an appeal committee, that way the members doing the certifying would not feel overwhelmed, and have to deal with some negativity stemming from disagreements?

This is kind of how my town works. If someone wants to build something and is denied, they go to the appeals board. My father serves on that board, but they only meet when someone has an appeal, he has no input into the potential rules of building in town.

As an example, one guy built a shed right next to the road, and people complained. The appeals board went to work and found out we do have a 70 foot setback from the road here, but it is measured from center of road to your front door. Is that right? Wrong? It does not matter to the appeals board. They did not come up with the rule, just noted it was a rule and the shed was allowed to stay because the mans front door was 70 feet back from the road.

A PEP appeals committee would do the same thing.

I suspect 70% of the appeals would come from people who don't have BB10 yet.  Based on a bit of experience with those types of submissions, I think they would all get rejected.  So I can see some reasoning to say you can't even appeal until you have BB10.

Mike Haasl wrote:

Anything that brings some coin to Paul to keep Permies going is a good thing.

Does Paul and the Permies site need money that badly to keep the place ticking? I'd hate to lose this forum, especially since the Kickstarters I've been a part of seem to raise so much more than is even generously needed for their purposes. This last one was phenomenal in the amount raised. Please correct me if my assumption is wrong, isn't the excess used towards the running of everything?

I'm guessing a little bit here but I believe Paul loses money on permies.com.  Kickstarters keep Wheaton Labs projects and permies.com afloat with influxes of cash from time to time.  It would be really nice if permies.com could support itself without needing help from other departments.

I just ran across a BB submitted by someone I don't know, which I think was judged wrong*. So I came searching for how to appeal and found this thread. While reading through it, I thought of a couple things:

1) What if you can't appeal your own? I'd be willing to 'sponsor' an appeal in the case of this one I just found as a disinterested third-party.

2) What if you had to gamble some BBV? BBV is a measure of work performed to make this system work, right? So if I have enough, maybe I know a little something about it and people take me more seriously than if I'm a sub BB10 participant as mentioned just above. So I put 5 (or whatever) BBV on the line -- this triggers an appeal and 3 judges who know the field of competence quickly discuss it, weighing the merit of what I'm saying, and if they agree, the judgement is overturned, and my BBV are safe. But if they don't, I lose (some/all) my ventured BBV -- the cost of proving that I don't know what's what. Maybe the specs are edited to make things more clear either way things go.

* "Wrong" in this case means technically right, I guess, but it's not even crystal clear at that. The submission was *much* more sophisticated than the requirements and four times the required size. Rejecting the submission did no good and might have put the submitter off of Permies. I even think the judge was sweet in the comments about the rejection, I just don't think it should have gone that way.

I take from the content of this thread, that there is no system in place for me to kick this mistake up to.

Christopher Weeks wrote:I just ran across a BB submitted by someone I don't know, which I think was judged wrong*. So I came searching for how to appeal and found this thread. While reading through it, I thought of a couple things:
...
I take from the content of this thread, that there is no system in place for me to kick this mistake up to.

Hi Christoper,
you are right, in that we are still figuring out how this might work.  Thanks for the suggestions on BBV and sponsoring.  I know that we have rejected some BBs because the person didn't follow the directions and this is a really important part of the program to Paul and potential Otis's.

A couple of examples come to mind:
1) someone harvested mint and submitted one photo saying they made tea and drank it right away when the BB says to harvest, dry, and store enough to make tea.
2) a shawl (in textiles) that was submitted in the scarf BB and while the shawl had more square inches than the required scarf it did not meet the specified size minimums

I always struggle with rejecting a BB and there is often discussion before a rejection.  I also feel it is important to hold everyone to a high and consistent standard.

I've thought about your response overnight. And I want to start by stating up front, I don't think I have any rights regarding this issue. If it goes nowhere and I don't get my way, I'm entirely comfortable with that reality and no hard feelings. But I got a masters degree with an emphasis in vocational educational testing, which, as you might imagine, comes up as a topic of conversation exceedingly infrequently. So more than anything, this is an interesting conversation to me.

Opalyn Rose wrote:2) a shawl (in textiles) that was submitted in the scarf BB and while the shawl had more square inches than the required scarf it did not meet the specified size minimums

I hope there's no harm going ahead and being explicit. This is the one that I think was judged wrong. And part of why I included the step in my suggested process of revising the text of the BB is because it doesn't say the scarf has to be 4" thick everywhere. It says "must be at least 4 inches wide". The scarf-shawl that was submitted is variable-width averaging 16.5 inches over the length. I acknowledge that one could interpret the spirit of the instructions the way you are doing, but it's certainly not the only reasonable way to do so. And since the submission doesn't really violate the exact written requirements and is so obviously way beyond the requirements imposed, I'm having a hard time seeing how anything (including Paul or Otis') are being helped by this miscarriage of justice. ;-)

(Actually, as I was typing that last bit above, it occurred to me that giving cover to future BB judges for being strict is a potential benefit. I could see the adjudication standards sliding toward laxity over time if there's no countervailing pressure keeping things honest. I've just recently rejected a couple of BB submissions on technicalities which I've been uncomfortable with previously, so I get that this is a real concern.)

This last note is off-topic from PEP Court, but I'm not sure where else to bring it up: but as I was writing the above, and thinking about all the ways that that submission "proved" the submitter met all the important criteria of the BB, it occurred to me how very easy it would be to just lift images from Ravelry (etc) and submit those as if they were my own. I guess cheating is a whole other can of worms and thinking about it is kind of depressing.

Thanks for reading. :)

Without seeing the BB in question, from what's said here, I would probably not approve that BB.  But I'm happy to review it if you can post a link.  

Scarves, especially knitted and crochet, tend to be a very specific thing where they are many times longer than the width.  Although it seems like a shawl would have within it a scarf, I find scarves more emotionally draining than knitting a shawl because the back and forth goes on forever.  I would rather knit an afghan or a shawl any day.  

And for me, the biggest attraction of SKIP isn't so much the skills they are doing, but the one ultimate skill they are showing - that they can follow instructions.  I've given up hiring or accepting help on the farm because no matter how many skills they have, I have yet to find anyone who can listen to the instructions.  They know better than me - because they read books (I read the same books, and my years of experience has sown me that those books are useless in the real world).  It ends up in dead animals and injured humans (usually me, often perminant).  At best, I get to spend twice the time fixing their 'help' than it would have taken me to do the task.  All because they won't (or never learned how to) follow instructions.   When I'm approving a BB, I look carefully at the instructions to see if they followed them to the letter.  

I find it helps to change the context.

Let's say the BB was to split and stack a pile of kindling 2 foot high, by 2 foot wide, and the sticks are 10 inches long.  

But the person submitted a cord (metric) of chopped wood.  A cord of wood contains within it a pile of kindling as potential.  The action is the same, only they did more of it.  It's a pile that exceeds the minimum requirements.  And there is surely the required amount of kindling inside that pile from all the little bits that fly off when wood meets axe.  But would that be approved for a badge of a pile of kindling?  

r ranson wrote:I'm happy to review it if you can post a link.

This is the submission.

Depending on how strictly you mean to find the scarf within the shawl, you may not. I think it's missing two 3"x4" right triangles but then has an additional 675 sq inches of shawl-point.

oh my goodness.  That is such a beautiful shawl.  The colours!  oh wow.

But definitely doesn't match the requirements.  

When making the textile BBs we had many hours of discussion of which get an area as the requirements and which have minimin dimensions to meet.  I think this one had almost two hours (times 5 people...).  Definitely, this BB is about meeting the dimensions.  Some of the reasons I gave above.

We don't have an oddball for textiles, but it could be submitted for the general oddball BB.  

Thanks for looking at it. I ran across this because I'll be finishing and submitting a scarf in the next week or so and I was reading through the thread. I completely believe that you had all that discussion and the consensus was that it has to be at least 4" wide along the entire 54" length. It would be neat if the text of the requirements actually said that. (My scarf will not be any kind of edge case. :-) )

I suspect based on her submission pattern that she was submitting documentation from projects she previously completed and had pictures for. (Which I think is entirely legit, yeah?) So it's not quite as heart-breaking as it would be if she had done that specifically to submit for this BB and because of vague wording, was turned away. That would be a real bummer to experience.

And for me, the biggest attraction of SKIP isn't so much the skills they are doing, but the one ultimate skill they are showing - that they can follow instructions. ...  All because they won't (or never learned how to) follow instructions.   When I'm approving a BB, I look carefully at the instructions to see if they followed them to the letter.  

A year later, I feel this even stronger.  For me it's about how they approach the task, not so much how they did on the task. Skills can easily be improved if the attitude is good.